README
1A basclass for audiovisualizers. Takes care of re-fitting the audio-rate to
2video-rate. It receives audio-data at the sampling-rate. It needs to render
3video-frames at frame-rate. The rendering needs n audio samples (depends on
4subclass). The baseclass takes care of that.
5
6Some effects could be enhanced by running geometrictransform/effecttc elements
7afterwards.
8
9= Feedback =
10* put 'Audio' to klass as well ?
11
12= API =
13
14* we have a couple of drawing helpers in gstdrawhelpers.h
15 (would be nice if we could use cairo)
16 draw_point (x,y,color);
17 draw_line (x1,x2,y1,y2,color);
18* some more we could add:
19 draw_hline (x1,x2,y,color);
20 draw_vline (x,y1,y2,color);
21 draw_rect (x1,x2,y1,y2,color);
22 draw_box (x1,x2,y1,y2,color); // filled
23* shading effects
24 - would be nice to use a generic 3x3 matrix operation, we don't run inplace
25 anyway
26 - this way we could also blur the background
27 - we need to handle visualizer that don't draw with alpha:
28 - add API to set can_mix and if so enable shaders
29 - ev. do mixing ourself (needs extra buffer)
30
31= Elements to port =
32gst-plugins-ugly/gst/synaestesia -> synaescope
33gst-plugins-bad/gst/smoothwave -> wavescope
34gst-plugins-good/gst/monoscope -> blend into what we have in wavescope
35- it keeps an array of the values we just showed last[width]
36- each audio-buffer is 2*width
37- it uses convolution to finde the best match in current[2*width] compared to
38 last[width]
39- then it shows current[match ... match+width] and copies it into last
40
41gst-plugins-base/ext/libvisual - done
42
43= Elements to add =
44spectrascope - done
45spacescope - stereo wavescope
46- left->x, right->y - done
47- polar mapping
48wavescope
49- we could have a bouncing line as a base, like a quix:
50 - two dots moving on a linear path and getting a new random dx,dy when hitting
51 a border
52 - the abs(dx/dy) - speed of movement - could be scaled by the sound level
53 - we would need to rotate, stretch and clip the waveform drawing to fit the
54 line
55 - we could scratch the rotate part and just stretch/squeeze x and shift/clip y
56
57xxxscope
58- have a matrix of source and drawing-functions
59 - sources: audio, spectrum, audio-low, audio-mid, audio-hi
60 - drawing: waves (style, color), space (style,color)
61- have the usual shade and move operations
62- have a way to draw each operator in one or more color-channels
63- we could calculate the sound-level (like in level element) and modulate
64 colors/movements
65 - for filtered low/mid/hi audio we could use different peak-falloffs
66
67= TODO =
68- element maker template
69- test for baseclass
70
71- actors
72 - we use the wave, filtered waves, balance and fft so far
73 - we could have narrow filters over harmonic frequencies
74 - we could use loudness like determined in level-meter
75
76- we probably want a VisBin like the gnome video effects
77- this way we can specify pipeline fragments
78- VisBin can use a videomixer to switch effects based on time or song
79- VisBin can e.g. control a text-overlay to render the title into the
80 visualisation for a while
81
82= Test it =
83
84GST_DEBUG="*:2,*scope*:4"
85
86GST_PLUGIN_PATH=$GST_PLUGIN_PATH:$PWD gst-inspect scopes
87
88== 0.10 ==
89GST_PLUGIN_PATH=$GST_PLUGIN_PATH:$PWD gst-launch audiotestsrc ! audioconvert ! wavescope ! colorspace ! ximagesink
90GST_PLUGIN_PATH=$GST_PLUGIN_PATH:$PWD gst-launch filesrc location=$HOME/Music/1.mp3 ! decodebin2 ! audioconvert ! wavescope ! colorspace ! ximagesink
91GST_PLUGIN_PATH=$GST_PLUGIN_PATH:$PWD gst-launch filesrc location=$HOME/Music/1.mp3 ! decodebin2 ! audioconvert ! wavescope style=lines shade-amount=0x00080402 ! edgetv ! vertigotv ! ximagesink
92
93GST_PLUGIN_PATH=$GST_PLUGIN_PATH:$PWD gst-launch filesrc location=$HOME/Music/1.mp3 ! decodebin2 ! audioconvert ! spacescope style=lines shade-amount=0x00080402 ! ximagesink
94GST_PLUGIN_PATH=$GST_PLUGIN_PATH:$PWD gst-launch filesrc location=$HOME/Music/1.mp3 ! decodebin2 ! audioconvert ! spacescope style=lines shade-amount=0x00080402 ! vertigotv ! ximagesink
95
96GST_PLUGIN_PATH=$GST_PLUGIN_PATH:$PWD gst-launch filesrc location=$HOME/Music/1.mp3 ! decodebin2 ! audioconvert ! spectrascope ! colorspace ! ximagesink
97GST_PLUGIN_PATH=$GST_PLUGIN_PATH:$PWD gst-launch filesrc location=$HOME/Music/1.mp3 ! decodebin2 ! audioconvert ! spectrascope shader=fade-and-move-up shade-amount=0x00040302 ! colorspace ! ximagesink
98
99GST_PLUGIN_PATH=$GST_PLUGIN_PATH:$PWD gst-launch filesrc location=$HOME/Music/1.mp3 ! decodebin2 ! tee name=t ! queue ! audioconvert ! synaesthesia ! ximagesink t. ! queue ! synaescope shade-amount=0x00040404 ! colorspace ! ximagesink
100
101== 0.11/1.0 ==
102gst-launch-1.0 audiotestsrc ! audioconvert ! wavescope ! videoconvert ! ximagesink
103
104gst-launch-1.0 filesrc location=$HOME/Music/1.mp3 ! decodebin ! audioconvert ! spectrascope ! videoconvert ! ximagesink
105
106gst-launch-1.0 filesrc location=$HOME/Music/1.mp3 ! decodebin ! tee name=t ! queue ! audioconvert ! wavescope style=color-lines shade-amount=0x00080402 ! alpha alpha=0.5 ! videomixer name=m background=black ! videoconvert ! vertigotv ! ximagesink t. ! queue ! audioconvert ! spacescope style=color-lines shade-amount=0x00080402 ! alpha alpha=0.5 ! m. t. ! queue ! autoaudiosink
107
108
109