Commit graph

79 commits

Author SHA1 Message Date
Romain Vimont
fbe0f951e1 Add audio player
Play the decoded audio using SDL.

The audio player frame sink receives the audio frames, resample them
and write them to a byte buffer (introduced by this commit).

On SDL audio callback (from an internal SDL thread), copy samples from
this byte buffer to the SDL audio buffer.

The byte buffer is protected by the SDL_AudioDeviceLock(), but it has
been designed so that the producer and the consumer may write and read
in parallel, provided that they don't access the same slices of the
ring-buffer buffer.

PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>

Co-authored-by: Simon Chan <1330321+yume-chan@users.noreply.github.com>
2023-03-10 22:22:15 +01:00
Romain Vimont
1230149fdd Use delay buffer as a frame source/sink
The components needing delayed frames (sc_screen and sc_v4l2_sink)
managed a sc_video_buffer instance, which itself embedded a
sc_frame_buffer instance (to keep only the most recent frame).

In theory, these components should not be aware of delaying: they should
just receive AVFrames later, and only handle a sc_frame_buffer.

Therefore, refactor sc_delay_buffer as a frame source (it consumes)
frames) and a frame sink (it produces frames, after some delay), and
plug an instance in the pipeline only when a delay is requested.

This also removes the need for a specific sc_video_buffer.

PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-03-10 22:22:15 +01:00
Romain Vimont
974227a3fc Use frame source trait in decoder 2023-03-10 22:22:15 +01:00
Romain Vimont
6e05d7047a Call avcodec_receive_frame() in a loop
Since in scrcpy a video packet passed to avcodec_send_packet() is always
a complete video frame, it is sufficient to call avcodec_receive_frame()
exactly once.

In practice, it also works for audio packets: the decoder produces
exactly 1 frame for 1 input packet.

In theory, it is an implementation detail though, so
avcodec_receive_frame() should be called in a loop.
2023-03-10 22:22:15 +01:00
Romain Vimont
619730edaf Pass AVCodecContext to frame sinks
Frame consumers may need details about the frame format.

PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-03-10 22:22:15 +01:00
Romain Vimont
05f0e35d2a Give a name to decoder instances
This will be useful in logs.

PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-03-10 22:22:15 +01:00
Romain Vimont
e02f30f895 Remove unnecessary error logs
When a call to a packet or frame sink fails, do not log the error on the
caller side: either the "failure" is expected (explicitly stopped) or it
must be logged by the packet or frame sink implementation.
2023-02-19 02:10:14 +01:00
Romain Vimont
0080d0b0ff Use sc_ prefix for decoder 2022-02-02 21:03:55 +01:00
Romain Vimont
09c55b0f93 Set "low delay" decoder flag
I don't really know the concrete benefits, but scrcpy definitely wants
low delay decoding.

Suggested-by: François Cartegnie <fcvlcdev@free.fr>
2021-12-08 23:53:54 +01:00
Romain Vimont
3653fb6b15 Add OutOfMemory log helper
Add a special LOG_OOM() function to log all OutOfMemory errors (i.e.
allocations returning NULL).
2021-11-24 22:06:49 +01:00
Romain Vimont
31131039bb Add missing includes
Refs #2616 <https://github.com/Genymobile/scrcpy/issues/2616>
2021-09-20 18:27:37 +02:00
Romain Vimont
6fd7e89da1 Explicitly initialize decoder sink_count
The zero-initialization relied on the fact that the decoder instance is
static.
2021-05-28 21:23:18 +02:00
Romain Vimont
ae6ec7a194 Unref decoder AVFrame immediately
The frame can be unref immediately after it is pushed to the frame
sinks.

It was not really a memory leak because the frame was unref every time
by avcodec_receive_frame() (and freed on close), but a reference was
unnecessarily kept for too long.
2021-04-26 18:05:43 +02:00
Romain Vimont
8b90dc61b9 Handle EAGAIN on send_packet in decoder
EAGAIN was only handled on receive_frame.

In practice, it should not be necessary, since one packet always
contains one frame. But just in case.
2021-04-25 14:38:42 +02:00
Romain Vimont
6f5ad21f57 Make decoder push frames to sinks
Now that screen implements the packet sink trait, make decoder push
packets to the sinks without depending on the concrete sink types.
2021-04-25 14:38:42 +02:00
Romain Vimont
f7a1b67d66 Make stream push packets to sinks
Now that decoder and recorder implement the packet sink trait, make
stream push packets to the sinks without depending on the concrete sink
types.
2021-04-25 14:38:42 +02:00
Romain Vimont
cbed38799e Expose decoder as packet sink
Make decoder implement the packet sink trait.

This will allow the stream to push packets without depending on the
concrete sink type.
2021-04-25 14:38:42 +02:00
Romain Vimont
5beb7d6c02 Reorder decoder functions
This will make further commits more readable.
2021-04-25 14:38:42 +02:00
Romain Vimont
2ddf760c09 Make video_buffer more generic
The video buffer took ownership of the producer frame (so that it could
swap frames quickly).

In order to support multiple sinks plugged to the decoder, the decoded
frame must not be consumed by the display video buffer.

Therefore, move the producer and consumer frames out of the video
buffer, and use FFmpeg AVFrame refcounting to share ownership while
avoiding copies.
2021-04-25 14:38:42 +02:00
Romain Vimont
de9b79ec2d Remove compat with old FFmpeg decoding API
The new API has been introduced in 2016 in libavcodec 57.xx, it's very
old.

This will avoid to maintain two code paths for decoding.
2021-04-25 14:38:42 +02:00
Romain Vimont
55806e7d31 Remove option --render-expired-frames
This flag forced the decoder to wait for the previous frame to be
consumed by the display.

It was initially implemented as a compilation flag for testing, not
intended to be exposed at runtime. But to remove ifdefs and to allow
users to test this flag easily, it had finally been exposed by commit
ebccb9f6cc.

In practice, it turned out to be useless: it had no practical impact,
and it did not solve or mitigate any performance issues causing frame
skipping.

But that added some complexity to the codebase: it required an
additional condition variable, and made video buffer calls possibly
blocking, which in turn required code to interrupt it on exit.

To prepare support for multiple sinks plugged to the decoder (display
and v4l2 for example), the blocking call used for pacing the decoder
output becomes unacceptable, so just remove this useless "feature".
2021-04-25 14:38:42 +02:00
Romain Vimont
a09733d175 Remove useless includes from decoder.c 2021-04-10 18:48:44 +02:00
Romain Vimont
6231f683af Fix compilation error for old decoding API
Commits cb9c42bdcb and
441d3fb119 updated the code only for the
new decoding API.
2021-04-05 22:35:14 +02:00
Romain Vimont
cb9c42bdcb Use a callback to notify frame skip
A skipped frame is detected when the producer offers a frame while the
current pending frame has not been consumed.

However, the producer (in practice the decoder) is not interested in the
fact that a frame has been skipped, only the consumer (the renderer) is.

Therefore, notify frame skip via a consumer callback. This allows to
manage the skipped and rendered frames count at the same place, and
remove fps_counter from decoder.
2021-03-06 22:58:03 +01:00
Romain Vimont
fb9f9848bd Use a callback to notify a new frame
Make the decoder independant of the SDL even mechanism, by making the
consumer register a callback on the video_buffer.
2021-03-06 22:58:03 +01:00
Romain Vimont
441d3fb119 Make video buffer more generic
Video buffer is a tool between a frame producer and a frame consumer.

For now, it is used between a decoder and a renderer, but in the future
another instance might be used to swscale decoded frames.
2021-03-06 22:58:03 +01:00
Romain Vimont
cb197ee3a2 Move fps counter out of video buffer
In order to make video buffer more generic, move out its specific
responsibility to count the fps between the decoder and the renderer.
2021-03-06 22:58:03 +01:00
Romain Vimont
f6320c7e31 Wrap SDL thread functions into scrcpy-specific API
The goal is to expose a consistent API for system tools, and paves the
way to make the "core" independant of SDL in the future.
2021-02-17 09:54:03 +01:00
Romain Vimont
59feb2a15c Group common includes into common.h
Include config.h and compat.h in common.h, and include common.h from all
source files.
2021-01-08 19:22:10 +01:00
Romain Vimont
d0f5a7fd9f Remove unused includes
No mutex is used in decoder.c and stream.c.
2019-11-27 21:40:54 +01:00
Romain Vimont
510caff0cd Replace SDL_assert() by assert()
SDL_assert() open a dialog on assertion failure.

There is no reason not to use assert() directly.
2019-11-27 21:19:46 +01:00
Romain Vimont
dfd0707a29 Move utilities to util/ 2019-11-24 11:53:23 +01:00
Romain Vimont
1f8ba1ca79 Include config.h everywhere
Ref: <https://github.com/Genymobile/scrcpy/issues/829>

Suggested-by: Louis Kruger <louisk@gmail.com>
2019-09-29 22:39:53 +02:00
Romain Vimont
a7b3901c31 Add more consts
Some decoder and recorder functions must not write to AVCodec and
AVPacket.
2019-03-03 12:02:41 +01:00
Romain Vimont
8595862005 Use explicit output parameter for skipped frame
The function video_buffer_offer_decoded_frame() returned a bool to
indicate whether the previous frame had been consumed.

This was confusing, because we could expect the returned bool report
whether the action succeeded.

Make the semantic explicit by using an output parameter.

Also revert the flag (report if the frame has been skipped instead of
consumed) to avoid confusion for the first frame (the previous is
neither skipped nor consumed because there is no previous frame).
2019-03-03 00:35:20 +01:00
Romain Vimont
dfed1b250e Replace SDL types by C99 standard types
Scrcpy is a C11 project. Use the C99 standard types instead of the
SDL-specific types:

    SDL_bool -> bool
    SintXX   -> intXX_t
    UintXX   -> uintXX_t
2019-03-02 23:55:23 +01:00
Romain Vimont
aeda583a2c Update code style
Limit source code to 80 chars, and declare functions return type and
modifiers on a separate line.

This allows to avoid very long lines, and all function names are
aligned.

(We do this on VLC, and I like it.)
2019-03-02 20:28:46 +01:00
Romain Vimont
e6e011baaf Add stream layer
The decoder initially read from the socket, decoded the video and sent
the decoded frames to the screen:

              +---------+      +----------+
  socket ---> | decoder | ---> |  screen  |
              +---------+      +----------+

The design was simple, but the decoder had several responsabilities.

Then we added the recording feature, so we added a recorder, which
reused the packets received from the socket managed by the decoder:

                                    +----------+
                               ---> |  screen  |
              +---------+     /     +----------+
  socket ---> | decoder | ----
              +---------+     \     +----------+
                               ---> | recorder |
                                    +----------+

This lack of separation of concerns now have concrete implications: we
could not (properly) disable the decoder/display to only record the
video.

Therefore, split the decoder to extract the stream:

                                    +----------+      +----------+
                               ---> | decoder  | ---> |  screen  |
              +---------+     /     +----------+      +----------+
  socket ---> | stream  | ----
              +---------+     \     +----------+
                               ---> | recorder |
                                    +----------+

This will allow to record the stream without decoding the video.
2019-03-02 18:45:45 +01:00
Romain Vimont
bcd4090d51 Fix recording with old decoding/encoding API
The deprecated avcodec_decode_video2() should always the whole packet,
so there is no need to loop (cf doc/examples/demuxing_decoding.c in
FFmpeg).

This hack changed the packet size and data pointer. This broke recording
which used the same packet.
2019-03-02 17:03:15 +01:00
Romain Vimont
84270e2d18 Rename "stop" to "interrupt"
The purpose of video_buffer_stop() is to interrupt any blocking call, so
rename it to video_buffer_interrupt().
2019-03-02 17:03:15 +01:00
Romain Vimont
fff87095d9 Rename "frames" to "video_buffer"
It better describes the purpose of the structure.
2019-03-02 15:24:33 +01:00
Romain Vimont
751600a7f9 Move all compat ifdefs definitions to compat.h
This allows to give a proper name to features requirements.
2019-02-16 15:28:56 +01:00
Romain Vimont
1a5ba59504 Fix memory leak on close
The buffer associated to the AVIOContext must be freed.
2019-02-09 12:27:13 +01:00
Romain Vimont
2876463d39 Fix read_packet() return value on error or EOF
Fix warning on error or EOF:

> Invalid return value 0 for stream protocol

See <http://git.videolan.org/?p=ffmpeg.git;a=commitdiff;h=a606f27f4c610708fa96e35eed7b7537d3d8f712>.

Fixes <https://github.com/Genymobile/scrcpy/issues/333>.
2018-11-12 14:45:43 +01:00
Romain Vimont
22ff03f2f7 Do not queue invalid PTS
Configuration packets produced by MediaCodec have no valid PTS, and do
not produce frame. Do not queue their (invalid) PTS not to break the
matching between frames and their PTS.
2018-11-11 21:37:31 +01:00
Romain Vimont
60afb46c8d Store queue of PTS for pending frames
Several frames may be read by read_packet() before they are consumed
(returned by av_read_frame()), so we need to store the PTS of frames in
order, so that the right PTS is assigned to the right frame.
2018-11-11 21:37:31 +01:00
Romain Vimont
345f8858d3 Send frame meta only if recording is enabled
The client needs the PTS for each frame only if recording is enabled.
Otherwise, the PTS are not necessary, and the protocol is more
straighforward.
2018-11-11 21:37:31 +01:00
Romain Vimont
70579dc709 Wrap receiver state into separate struct
For readability, wrap the state of the receiver in a separate struct
receiver_state.
2018-11-11 21:37:31 +01:00
Romain Vimont
e562837c0b Avoid partial header reads
Use net_recv_all() to avoid partial reads for the "meta" header (this
would break the whole stream).
2018-11-11 21:37:31 +01:00
Romain Vimont
ebe998cf78 Move buffer reader functions to buffer_util.h 2018-11-11 21:37:31 +01:00