Skip to content

Commit

Permalink
first commit
Browse files Browse the repository at this point in the history
  • Loading branch information
@ColumPaget
ColumPaget committed Feb 9, 2014
0 parents commit 53d49b8
Show file tree
Hide file tree
Showing 149 changed files with 45,878 additions and 0 deletions.
92 changes: 92 additions & 0 deletions CHANGES
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
Version 1.2.1
Fixes for many sites.
Added 'sshtunnel' proxy type

Version 1.2.0
Removed 'Clipshack' (Dead Site)
Removed '5 min' (better dead than AOL)
Removed 'PressTV' (Switched to rtmp? In this day and age?)
Added 'Funny or die'
Fixes for most sites
Much work done on 'ipad streaming' downloads (.m3u8 files)


Version 1.1.15
Added a basic 'ETA' meter (bounces around a lot depending on speed of download)
Added support form links from google-video search
Fixed dailymotion titles

Version 1.1.14
Largefile (> 2GB) support
Added 'Animehere' website

Version 1.1.13
Fixed Academic Earth.
Fixed Teachertube.
Fixed Escapist Magazine.

Added "International Business Times",
Added "Sidney Morning Herald",
Added "Press TV (Iran)",
Added www.videojug.com

Removed Dead Site: VideosFacebook

Added support for youtube /embed/ format links

Version 1.1.12
Big thank you to Abhisek Sanyal for submitting a patch that fixed broken youtube!
Thanks to 'nibbles' for a bunch of bugfixes that let movgrab compile under OSX

Version 1.1.11
Fixes for various sites

Version 1.1.10
Fixed '-w' flag, which wasn't working
Added 'National Geographic'
Added 'Videobash'

Version 1.1.9
Fixes for mefeedia and metatube
Added filesizes to 'Available Formats' list

Version 1.1.8
Added crazy new youtube formats like 'webm-3D'.
Movgrab now honors the 'http_proxy' environment variable

Version 1.1.7
Added Bloomberg
Added Discovery Channel
Fixed Mobango
Fixed Redbalcony
Assorted bugfixes

Version 1.1.6
Fixed Dailymotion
Youtube more reliable
'help' now goes to stdout

Version 1.1.5
Fixed TED
Fixed Dailymotion

Version 1.1.4
Makefile bugfix
Other minor bugfixes
Seen working on OSX

Version 1.1.3
Added -r flag for resuming downloads
Added 'TeachTube' downloads
MacOSX segfault fixed (hopefully)

Version 1.1.2:
Vimeo working again
Debianized Makefile

Version 1.1.1:
Various bugfixes.
Removed '-s' and '-sc', which related to previous streaming methods, and replaced with '-P'. '-P <program>' allows you to specify a 'player program' that will be used to play the downloaded file. By default this will launch after 25% of the file is downloaded, but this value can be set with '-Pp <percent>'.
Added '+o' flag, which allows additional files for writing a download to, so that you can stream it and record it at the same time.


71 changes: 71 additions & 0 deletions Docs/HowToAddSites.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
Everything relating to adding sites is now in the files 'servicetypes.c' and 'servicetypes.h'.

servicetypes.h contains an enumerated type of values that defines a TYPE_ for each download type. You must add a TYPE_ to this too.

Everything else is in 'servicetypes.c'. Near the top of this file are 3 arrays of strings:

DownloadTypes
DownloadNames
TestLinks

When you add a site, you add an entry to each of these. A 'type code' that is used to select it from the command line with the -t option, a longer 'DownloadName' for the download type/website and a link to a video page that can be used to test if this download method is working. All three must be added to the same place in their respective arrays, so if you insert a new download type as the tenth type, then it's entries must be at position ten in all these arrays.


Now find a function called 'ExtractItemInfo'. It contains a massive switch statement, with an entry for each of the download TYPE_ values. Most have a format like this:

case TYPE_MYVIDEO:
#define MYVIDEO_URL_START "link rel='image_src' href='"
#define MYVIDEO_URL_END "/thumbs"
#define MYVIDEO_VIDID_END "_"

if (strstr(Tempstr,MYVIDEO_URL_START))
{
ptr=GenericExtractFromLine(Tempstr, "MyVidURL",MYVIDEO_URL_START,MYVIDEO_URL_END,Vars,EXTRACT_DEQUOTE
| EXTRACT_NOSPACES);
ptr=GenericExtractFromLine(ptr, "item:flv","/",MYVIDEO_VIDID_END,Vars,EXTRACT_DEQUOTE | EXTRACT_NOSPA
CES);
}


if (strstr(Tempstr,GENERIC_TITLE_START))
{
GenericExtractFromLine(Tempstr, "Title",GENERIC_TITLE_START,GENERIC_TITLE_END,Vars,EXTRACT_DEQUOTE);
}
break;


They call the function 'GenericExtractFromLine' to clip out bits of text. These are then added to the variable list 'Vars' with the name given as the second argument. These variables can then be accessed from the list with 'GetVar' or with 'SubstituteVarsInString'. GenericExtractFromLine is normally used to clip serveral values from a webpage, that can be used in the actual download later.

The 'Title' variable is generally used to name the downloaded file. The 'item' variables contain the final download url for the movie. The following 'item' variables can exist. 'item:flv', 'item:mp4', 'item:mov', 'item:mpg', 'item:wmv', 'item:w4a', 'item:avi', 'item:3gp' 'item:mp3', 'item:m4a'. There is also the special value 'item:reference', which is used to tell the system not to download the link as a video, but rather to get it as a webpage, and go through the process of ExtractItemInfo again.

Other variables have meaning on a download-type by download-type basis.

After 'ExtractItemInfo' has done its thing we should have a list of Variables that are used in 'GetNextURL'. This function is another switch statement, containing entries of the form:

case TYPE_MYVIDEO:
Tempstr=SubstituteVarsInString(Tempstr,"$(MyVidURL)/$(ID).flv",Vars,0);
RetVal=DownloadItem(Tempstr, Title,Post);
break;


The 'SubstituteVarsInString' fuction is used to construct a video URL from all the vars that were clipped out in 'ExtractItemInfo'. The 'ID' variable is a special case, it is the final selection from the 'item:' variables that were found on the website. Some websites might have many 'item' variables of different formats and video qualities, so we might have item:flv item:mp4 and item:wmv available for the same movie. The 'SelectDownloadFormat' function chooses one of these based on command-line options, and sets the 'ID' variable to have the same value.

'DownloadItem' is the final function that uses all this information to actually download the video.


One last thing remains. With all this work done, movgrab should now be able to download from your website using:

movgrab -t <type> <url>

However, it's a good idea to add an entry to 'IdentifyServiceType'. This function checks for strings in the video website URL to decide which type of download we're dealing with. It contains entries like:

else if (strstr(Server,"myvideo"))
{
Type=TYPE_MYVIDEO;
}


And with that added, movgrab should now be able to download videos just from the appropriate url.



14 changes: 14 additions & 0 deletions Docs/Proxys.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
Movgrab currently supports http and ssltunnel proxys. These an either be set using the -p/-proxy command-line switches, or using the http_proxy and sshtunnel_proxy environment variables.

proxy URLs have the form:

<type>://<username>:<password>@<host>:<port>

If no authentication is needed to use the proxy then the '<username>:<password>@' section can be ommited.

'type' can be:
http A standard http proxy
https An http proxy that uses encrypted connections
ssltunnel An ssh server that supports connection forwarding

All connections will then be forwarded through the proxy.
173 changes: 173 additions & 0 deletions Docs/Radio.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
As some of the video download sites that movgrab accesses use the 'IPad Streaming' technique, which involves the use of the .m3u container file, I decided to include support for container files in movgrab.

The main use for this is internet radio stations. These generally advertise themselves with a .m3u, .pls or .asx file, you download the file and within the file are details of where the radio stream can be found on the net.

Media players like mplayer can generally handle such streams without any assistance, however some players have less support than others. Also I have found that even mplayer and mpg123 have trouble with some streams, for example: http://www.triplag.com/webradio/darkpsy/triplag-darkpsy-playlist.asx

Using movgrab's "-o -" flag to stream the download to stdout, and then piping it into the player allows these problems to be overcome. So:

movgrab -o - 'http://www.triplag.com/webradio/darkpsy/triplag-darkpsy-playlist.asx' | mplayer - -cache 1024

Will get you your dark trance fix even if mplayer balks at the .asx file. Recent versions of mplayer require a -cache argument to stream from stdin. Even with this argument they complain about the cache emptying, but if you use a -cache it will eventually start to play.

Similarly mpg123 can be used thusly

movgrab -o - 'http://www.triplag.com/webradio/darkpsy/triplag-darkpsy-playlist.asx' | mpg123 -

For ogg-vorbis streams there's "ogg123" which comes as part of the vorbis distrobution.

Of course, movgrab can also write the stream to a file,

movgrab 'http://www.triplag.com/webradio/darkpsy/triplag-darkpsy-playlist.asx' -o darktrance.mp3

Best of all, a flag in the form '+o <path>' can be used to output the stream to a second (or third, or forth...) file, allowing you to listen to the stream and record it, thusly

movgrab 'http://www.triplag.com/webradio/darkpsy/triplag-darkpsy-playlist.asx' -o - +o darktrance.mp3 | mpg123 -

This both pipes it into 'mpg123' and writes it to 'darktrance.mp3'





RADIO STATIONS:

There's a Europe wide list at http://www.listenlive.eu/index.html

Here are some of the highlights of it

BBC world service. mpg123 doesn't seem to like this .pls, but movgrab can handle it and pipe the output into mpg123
http://www.bbc.co.uk/worldservice/meta/tx/nb/live/eneuk.pls

Chill - chillout/lounge
http://media-ice.musicradio.com/ChillMP3.m3u

Kerrang - Rock
http://tx.whatson.com/icecast.php?i=kerrang.mp3.m3u

Planet Rock - Rock
http://sharpflow.sharp-stream.com:8000/planetrock.mp3.m3u

Total Rock
http://icecast.playlouder.com:8000/totalrock.m3u

XFM - Rock and alternative
http://media-ice.musicradio.com/XFMMP3.m3u

Classic FM - Classical music
http://www.classicfm.com/

French Radio London
http://icy-e-01.sharp-stream.com/frenchradiolon.mp3.m3u



Trance/Dance Stations:

I've been listening to a lot of psy-trance recently (John OO Fleming's "Global Trance Grooves" podcast in particular) so there's a lot of these!


ETN.fm
http://etn.fm/playlists/etn1-mp3-medium.m3u - trace programme
http://etn.fm/playlists/etn2-mp3-medium.m3u - house programme

Club Lounge, mostly nice, lazy trance
http://www.club-lounge-radio.com/Club-Lounge-Radio.pls

Schizoid Radio, Broadcasting out of india, have 4 nice channels
http://schizoid.in/schizoid-chill.pls - Chillout
http://schizoid.in/schizoid-prog.pls - Progressive Trance
http://schizoid.in/schizoid-psy.pls - Psy-trance
http://schizoid.in/schizoid-edm.pls - General electronic dance music

French trance, more of a harder, techno sound
http://www.paris-one.com/pls/radio_trance.pls

Techno.fm
http://techno.fm/m3u/techno.mp3.m3u - Techno programme
http://techno.fm/m3u/trance.mp3.m3u - Trance programme

Trance moon, dark psy-trance
http://www.trancemoon.com/DARK.pls

http://89.238.166.195:9162/listen.pls
http://178.32.57.58:8382/listen.pls
http://london02.discovertrance.com:80/listen.pls
http://www.pulsradio.com/pls/pulsradio.pls
http://files.hard.fm/128.pls

Digitally Imported has a lot of channels for different types of electronic music

Chiptunes, music made with old computers and gaming consoles. Whatever will they think of next?
http://listen.di.fm/public3/chiptunes.pls

Spacemusic channel, very, very dreamy ambient
http://listen.di.fm/public3/spacemusic.pls

Maybe a few more BPM than spacemusic, but not much
http://listen.di.fm/public3/ambient.pls

Not quite as drugged out as spacemusic, but still lazy
http://listen.di.fm/public3/chillout.pls
http://listen.di.fm/public3/chilloutdreams.pls

#and a little more energetic
http://listen.di.fm/public3/lounge.pls
http://listen.di.fm/public3/soulfulhouse.pls

#Sound of the early 90's
http://listen.di.fm/public3/oldschoolhouse.pls

Chillout with more of an edgy sound, more psychadelic, alien or computery
http://listen.di.fm/public3/psychill.pls

I still don't know what 'progressive' means here
http://listen.di.fm/public3/progressive.pls

#more vocals, more dancy
http://listen.di.fm/public3/vocaltrance.pls
http://listen.di.fm/public3/eurodance.pls
http://listen.di.fm/public3/classiceurodance.pls
http://listen.di.fm/public3/club.pls
http://listen.di.fm/public3/house.pls
http://listen.di.fm/public3/discohouse.pls
http://listen.di.fm/public3/funkyhouse.pls

Arrrrriba!!!
http://listen.di.fm/public3/latinhouse.pls

Here come the drums!
http://listen.di.fm/public3/tribalhouse.pls

Dance-oriented with lots of effects and samples
http://listen.di.fm/public3/trance.pls
http://listen.di.fm/public3/classictrance.pls
http://listen.di.fm/public3/techno.pls
http://listen.di.fm/public3/electro.pls
http://listen.di.fm/public3/techhouse.pls
http://listen.di.fm/public3/minimal.pls
http://listen.di.fm/public3/classictechno.pls

Up to fifth gear. High BPM sci-fi dance
http://listen.di.fm/public3/goapsy.pls

#beats a little too fast to dance to really
http://listen.di.fm/public3/liquiddnb.pls
http://listen.di.fm/public3/drumandbass.pls

Breakbeat uses complex rhythmn arrangements
http://listen.di.fm/public3/breaks.pls

http://listen.di.fm/public3/futuresynthpop.pls

Grimy, dark and mechanical, straight outta Croydon!
http://listen.di.fm/public3/dubstep.pls


Insanely fast
http://listen.di.fm/public3/hardcore.pls
http://listen.di.fm/public3/harddance.pls

Dance music for skinheads. Simply horrible.
http://listen.di.fm/public3/gabber.pls
http://listen.di.fm/public3/hardstyle.pls
Loading

0 comments on commit 53d49b8

Please sign in to comment.