Android/scrcpy
1
0
Fork 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity
scrcpy/app/src/demuxer.c

311 lines
8.4 KiB
C
Raw Normal View History

Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
#include "demuxer.h"
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
Replace SDL_assert() by assert() SDL_assert() open a dialog on assertion failure. There is no reason not to use assert() directly.
2019-11-28 04:11:40 +08:00
#include <assert.h>
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
#include <libavutil/time.h>
#include <unistd.h>
#include "decoder.h"
#include "events.h"
Extract packet merging Config packets must be prepended to the next media packet. Extract the logic to a new sc_packet_merger helper to simplify the demuxer code.
2023-02-11 05:52:40 +08:00
#include "packet_merger.h"
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
#include "recorder.h"
Rename buffer_util.h to binary.h It will allow to expose more binary util functions not related to buffers. PR #3369 <https://github.com/Genymobile/scrcpy/pull/3369>
2022-08-03 21:13:16 +08:00
#include "util/binary.h"
Move utilities to util/
2019-11-24 18:53:00 +08:00
#include "util/log.h"
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
Rename HEADER_SIZE to SC_PACKET_HEADER_SIZE Prefix the constant for consistency.
2022-02-12 04:34:58 +08:00
#define SC_PACKET_HEADER_SIZE 12
Store packet flags in PTS most significant bits A special PTS value was used to encode a config packet. To prepare for adding more flags, use the most significant bits of the PTS field to store flags.
2022-02-12 02:31:01 +08:00
#define SC_PACKET_FLAG_CONFIG (UINT64_C(1) << 63)
Pass key frame flag from the device MediaCodec indicates when a packet is a key frame. Transmit it to the client.
2022-02-12 02:33:35 +08:00
#define SC_PACKET_FLAG_KEY_FRAME (UINT64_C(1) << 62)
Store packet flags in PTS most significant bits A special PTS value was used to encode a config packet. To prepare for adding more flags, use the most significant bits of the PTS field to store flags.
2022-02-12 02:31:01 +08:00
Pass key frame flag from the device MediaCodec indicates when a packet is a key frame. Transmit it to the client.
2022-02-12 02:33:35 +08:00
#define SC_PACKET_PTS_MASK (SC_PACKET_FLAG_KEY_FRAME - 1)
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
Add option to select video codec Introduce the selection mechanism. Alternative codecs will be added in further commits. PR #3713 <https://github.com/Genymobile/scrcpy/pull/3713>
2023-02-03 19:35:37 +08:00
static enum AVCodecID
Split codec_id reading Receive codec id and convert it to AVCodecID separately. This will allow the caller to distinguish between EOS and unknown codec id.
2023-02-11 01:48:20 +08:00
sc_demuxer_to_avcodec_id(uint32_t codec_id) {
Add option to select video codec Introduce the selection mechanism. Alternative codecs will be added in further commits. PR #3713 <https://github.com/Genymobile/scrcpy/pull/3713>
2023-02-03 19:35:37 +08:00
#define SC_CODEC_ID_H264 UINT32_C(0x68323634) // "h264" in ASCII
Add support for H265 Add option --codec=h265. PR #3713 <https://github.com/Genymobile/scrcpy/pull/3713> Fixes #3092 <https://github.com/Genymobile/scrcpy/issues/3092>
2023-02-03 19:40:55 +08:00
#define SC_CODEC_ID_H265 UINT32_C(0x68323635) // "h265" in ASCII
Add support for AV1 Add option --codec=av1. PR #3713 <https://github.com/Genymobile/scrcpy/pull/3713>
2023-02-03 19:42:22 +08:00
#define SC_CODEC_ID_AV1 UINT32_C(0x00617631) // "av1" in ASCII
Add an audio demuxer Add a demuxer which will read the stream from the audio socket. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-06 17:33:47 +08:00
#define SC_CODEC_ID_OPUS UINT32_C(0x6f707573) // "opus" in ASCII
Add option to select video codec Introduce the selection mechanism. Alternative codecs will be added in further commits. PR #3713 <https://github.com/Genymobile/scrcpy/pull/3713>
2023-02-03 19:35:37 +08:00
switch (codec_id) {
case SC_CODEC_ID_H264:
return AV_CODEC_ID_H264;
Add support for H265 Add option --codec=h265. PR #3713 <https://github.com/Genymobile/scrcpy/pull/3713> Fixes #3092 <https://github.com/Genymobile/scrcpy/issues/3092>
2023-02-03 19:40:55 +08:00
case SC_CODEC_ID_H265:
return AV_CODEC_ID_HEVC;
Add support for AV1 Add option --codec=av1. PR #3713 <https://github.com/Genymobile/scrcpy/pull/3713>
2023-02-03 19:42:22 +08:00
case SC_CODEC_ID_AV1:
return AV_CODEC_ID_AV1;
Add an audio demuxer Add a demuxer which will read the stream from the audio socket. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-06 17:33:47 +08:00
case SC_CODEC_ID_OPUS:
return AV_CODEC_ID_OPUS;
Add option to select video codec Introduce the selection mechanism. Alternative codecs will be added in further commits. PR #3713 <https://github.com/Genymobile/scrcpy/pull/3713>
2023-02-03 19:35:37 +08:00
default:
LOGE("Unknown codec id 0x%08" PRIx32, codec_id);
return AV_CODEC_ID_NONE;
}
}
Split codec_id reading Receive codec id and convert it to AVCodecID separately. This will allow the caller to distinguish between EOS and unknown codec id.
2023-02-11 01:48:20 +08:00
static bool
sc_demuxer_recv_codec_id(struct sc_demuxer *demuxer, uint32_t *codec_id) {
uint8_t data[4];
ssize_t r = net_recv_all(demuxer->socket, data, 4);
if (r < 4) {
return false;
}
*codec_id = sc_read32be(data);
return true;
}
Replace SDL types by C99 standard types Scrcpy is a C11 project. Use the C99 standard types instead of the SDL-specific types: SDL_bool -> bool SintXX -> intXX_t UintXX -> uintXX_t
2019-03-03 06:52:22 +08:00
static bool
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
sc_demuxer_recv_packet(struct sc_demuxer *demuxer, AVPacket *packet) {
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
// The video stream contains raw packets, without time information. When we
// record, we retrieve the timestamps separately, from a "meta" header
// added by the server before each raw packet.
//
// The "meta" header length is 12 bytes:
// [. . . . . . . .|. . . .]. . . . . . . . . . . . . . . ...
// <-------------> <-----> <-----------------------------...
// PTS packet raw packet
// size
//
// It is followed by <packet_size> bytes containing the packet/frame.
Store packet flags in PTS most significant bits A special PTS value was used to encode a config packet. To prepare for adding more flags, use the most significant bits of the PTS field to store flags.
2022-02-12 02:31:01 +08:00
//
// The most significant bits of the PTS are used for packet flags:
//
// byte 7 byte 6 byte 5 byte 4 byte 3 byte 2 byte 1 byte 0
Pass key frame flag from the device MediaCodec indicates when a packet is a key frame. Transmit it to the client.
2022-02-12 02:33:35 +08:00
// CK...... ........ ........ ........ ........ ........ ........ ........
// ^^<------------------------------------------------------------------->
// || PTS
Fix protocol documentation in comments Flags were in the correct order in the schema, but their description were reversed.
2022-07-20 17:39:55 +08:00
// | `- key frame
// `-- config packet
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
Rename HEADER_SIZE to SC_PACKET_HEADER_SIZE Prefix the constant for consistency.
2022-02-12 04:34:58 +08:00
uint8_t header[SC_PACKET_HEADER_SIZE];
ssize_t r = net_recv_all(demuxer->socket, header, SC_PACKET_HEADER_SIZE);
if (r < SC_PACKET_HEADER_SIZE) {
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
return false;
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
}
Rename buffer util functions with sc_ prefix
2022-02-12 16:12:46 +08:00
uint64_t pts_flags = sc_read64be(header);
uint32_t len = sc_read32be(&header[8]);
Replace SDL_assert() by assert() SDL_assert() open a dialog on assertion failure. There is no reason not to use assert() directly.
2019-11-28 04:11:40 +08:00
assert(len);
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
if (av_new_packet(packet, len)) {
Add OutOfMemory log helper Add a special LOG_OOM() function to log all OutOfMemory errors (i.e. allocations returning NULL).
2021-11-25 05:06:11 +08:00
LOG_OOM();
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
return false;
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
}
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
r = net_recv_all(demuxer->socket, packet->data, len);
Fix warnings on Windows fix warnings reported with -Dwarning_level=2 on Windows.
2019-11-27 20:39:42 +08:00
if (r < 0 || ((uint32_t) r) < len) {
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
av_packet_unref(packet);
return false;
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
}
Store packet flags in PTS most significant bits A special PTS value was used to encode a config packet. To prepare for adding more flags, use the most significant bits of the PTS field to store flags.
2022-02-12 02:31:01 +08:00
if (pts_flags & SC_PACKET_FLAG_CONFIG) {
packet->pts = AV_NOPTS_VALUE;
} else {
packet->pts = pts_flags & SC_PACKET_PTS_MASK;
}
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
Pass key frame flag from the device MediaCodec indicates when a packet is a key frame. Transmit it to the client.
2022-02-12 02:33:35 +08:00
if (pts_flags & SC_PACKET_FLAG_KEY_FRAME) {
packet->flags |= AV_PKT_FLAG_KEY;
}
Remove sc_demuxer_parse() Now that the key frame flag is known, parsing the packet is useless.
2022-02-12 02:35:47 +08:00
packet->dts = packet->pts;
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
return true;
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
}
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
static bool
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
push_packet_to_sinks(struct sc_demuxer *demuxer, const AVPacket *packet) {
for (unsigned i = 0; i < demuxer->sink_count; ++i) {
struct sc_packet_sink *sink = demuxer->sinks[i];
Make stream push packets to sinks Now that decoder and recorder implement the packet sink trait, make stream push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
if (!sink->ops->push(sink, packet)) {
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
return false;
}
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
}
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
return true;
}
static bool
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
sc_demuxer_push_packet(struct sc_demuxer *demuxer, AVPacket *packet) {
Simplify demuxer Call the same push_packet_to_sinks() in all cases, and make sc_demuxer_parse() return void.
2022-02-03 04:03:24 +08:00
bool ok = push_packet_to_sinks(demuxer, packet);
if (!ok) {
Give a name to demuxer instances This will be useful in logs. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-19 07:13:54 +08:00
LOGE("Demuxer '%s': could not process packet", demuxer->name);
Simplify demuxer Call the same push_packet_to_sinks() in all cases, and make sc_demuxer_parse() return void.
2022-02-03 04:03:24 +08:00
return false;
}
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
return true;
}
Make stream push packets to sinks Now that decoder and recorder implement the packet sink trait, make stream push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
static void
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
sc_demuxer_close_first_sinks(struct sc_demuxer *demuxer, unsigned count) {
Make stream push packets to sinks Now that decoder and recorder implement the packet sink trait, make stream push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
while (count) {
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
struct sc_packet_sink *sink = demuxer->sinks[--count];
Make stream push packets to sinks Now that decoder and recorder implement the packet sink trait, make stream push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
sink->ops->close(sink);
}
}
static inline void
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
sc_demuxer_close_sinks(struct sc_demuxer *demuxer) {
sc_demuxer_close_first_sinks(demuxer, demuxer->sink_count);
Make stream push packets to sinks Now that decoder and recorder implement the packet sink trait, make stream push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
}
static bool
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
sc_demuxer_open_sinks(struct sc_demuxer *demuxer, const AVCodec *codec) {
for (unsigned i = 0; i < demuxer->sink_count; ++i) {
struct sc_packet_sink *sink = demuxer->sinks[i];
Make stream push packets to sinks Now that decoder and recorder implement the packet sink trait, make stream push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
if (!sink->ops->open(sink, codec)) {
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
sc_demuxer_close_first_sinks(demuxer, i);
Make stream push packets to sinks Now that decoder and recorder implement the packet sink trait, make stream push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
return false;
}
}
return true;
}
Disable audio on initialization error By default, audio is enabled (--no-audio must be explicitly passed to disable it). However, some devices may not support audio capture (typically devices below Android 11, or Android 11 when the shell application is not foreground on start). In that case, make the server notify the client to dynamically disable audio forwarding so that it does not wait indefinitely for an audio stream. Also disable audio on unknown codec or missing decoder on the client-side, for the same reasons. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-19 01:09:18 +08:00
static void
sc_demuxer_disable_sinks(struct sc_demuxer *demuxer) {
for (unsigned i = 0; i < demuxer->sink_count; ++i) {
struct sc_packet_sink *sink = demuxer->sinks[i];
if (sink->ops->disable) {
sink->ops->disable(sink);
}
}
}
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
static int
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
run_demuxer(void *data) {
struct sc_demuxer *demuxer = data;
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
Properly report demuxer errors All demuxer errors were reported as "device disconnected", even if the failure was not related to device socket read.
2023-02-11 01:54:46 +08:00
// Flag to report end-of-stream (i.e. device disconnected)
bool eos = false;
Split codec_id reading Receive codec id and convert it to AVCodecID separately. This will allow the caller to distinguish between EOS and unknown codec id.
2023-02-11 01:48:20 +08:00
uint32_t raw_codec_id;
bool ok = sc_demuxer_recv_codec_id(demuxer, &raw_codec_id);
if (!ok) {
Disable audio on initialization error By default, audio is enabled (--no-audio must be explicitly passed to disable it). However, some devices may not support audio capture (typically devices below Android 11, or Android 11 when the shell application is not foreground on start). In that case, make the server notify the client to dynamically disable audio forwarding so that it does not wait indefinitely for an audio stream. Also disable audio on unknown codec or missing decoder on the client-side, for the same reasons. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-19 01:09:18 +08:00
LOGE("Demuxer '%s': stream disabled due to connection error",
demuxer->name);
eos = true;
goto end;
}
if (raw_codec_id == 0) {
LOGW("Demuxer '%s': stream explicitly disabled by the device",
demuxer->name);
sc_demuxer_disable_sinks(demuxer);
Properly report demuxer errors All demuxer errors were reported as "device disconnected", even if the failure was not related to device socket read.
2023-02-11 01:54:46 +08:00
eos = true;
Split codec_id reading Receive codec id and convert it to AVCodecID separately. This will allow the caller to distinguish between EOS and unknown codec id.
2023-02-11 01:48:20 +08:00
goto end;
}
enum AVCodecID codec_id = sc_demuxer_to_avcodec_id(raw_codec_id);
Add option to select video codec Introduce the selection mechanism. Alternative codecs will be added in further commits. PR #3713 <https://github.com/Genymobile/scrcpy/pull/3713>
2023-02-03 19:35:37 +08:00
if (codec_id == AV_CODEC_ID_NONE) {
Disable audio on initialization error By default, audio is enabled (--no-audio must be explicitly passed to disable it). However, some devices may not support audio capture (typically devices below Android 11, or Android 11 when the shell application is not foreground on start). In that case, make the server notify the client to dynamically disable audio forwarding so that it does not wait indefinitely for an audio stream. Also disable audio on unknown codec or missing decoder on the client-side, for the same reasons. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-19 01:09:18 +08:00
LOGE("Demuxer '%s': stream disabled due to unsupported codec",
demuxer->name);
sc_demuxer_disable_sinks(demuxer);
Add option to select video codec Introduce the selection mechanism. Alternative codecs will be added in further commits. PR #3713 <https://github.com/Genymobile/scrcpy/pull/3713>
2023-02-03 19:35:37 +08:00
goto end;
}
const AVCodec *codec = avcodec_find_decoder(codec_id);
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
if (!codec) {
Disable audio on initialization error By default, audio is enabled (--no-audio must be explicitly passed to disable it). However, some devices may not support audio capture (typically devices below Android 11, or Android 11 when the shell application is not foreground on start). In that case, make the server notify the client to dynamically disable audio forwarding so that it does not wait indefinitely for an audio stream. Also disable audio on unknown codec or missing decoder on the client-side, for the same reasons. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-19 01:09:18 +08:00
LOGE("Demuxer '%s': stream disabled due to missing decoder",
demuxer->name);
sc_demuxer_disable_sinks(demuxer);
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
goto end;
}
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
if (!sc_demuxer_open_sinks(demuxer, codec)) {
Remove unused codec context The demuxer does not need any codec context.
2023-02-11 01:45:31 +08:00
goto end;
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
}
Do not merge config audio packets For video streams (at least H.264 and H.265), the config packet containing SPS/PPS must be prepended to the next packet (the following keyframe). For audio streams (at least OPUS), they must not be merged. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-19 00:37:58 +08:00
// Config packets must be merged with the next non-config packet only for
// video streams
bool must_merge_config_packet = codec->type == AVMEDIA_TYPE_VIDEO;
Extract packet merging Config packets must be prepended to the next media packet. Extract the logic to a new sc_packet_merger helper to simplify the demuxer code.
2023-02-11 05:52:40 +08:00
struct sc_packet_merger merger;
Do not merge config audio packets For video streams (at least H.264 and H.265), the config packet containing SPS/PPS must be prepended to the next packet (the following keyframe). For audio streams (at least OPUS), they must not be merged. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-19 00:37:58 +08:00
if (must_merge_config_packet) {
sc_packet_merger_init(&merger);
}
Extract packet merging Config packets must be prepended to the next media packet. Extract the logic to a new sc_packet_merger helper to simplify the demuxer code.
2023-02-11 05:52:40 +08:00
Allocate AVPacket for local stream packet From FFmpeg/doc/APIchanges: 2021-03-17 - f7db77bd87 - lavc 58.133.100 - codec.h Deprecated av_init_packet(). Once removed, sizeof(AVPacket) will no longer be a part of the public ABI. Refs #2302 <https://github.com/Genymobile/scrcpy/issues/2302>
2021-06-14 15:07:49 +08:00
AVPacket *packet = av_packet_alloc();
if (!packet) {
Add OutOfMemory log helper Add a special LOG_OOM() function to log all OutOfMemory errors (i.e. allocations returning NULL).
2021-11-25 05:06:11 +08:00
LOG_OOM();
Remove unused parser Since 1c02b58412db2646ba6e57a7e305d26482b9bc0d, the parser is not used anymore.
2023-02-11 06:18:09 +08:00
goto finally_close_sinks;
Allocate AVPacket for local stream packet From FFmpeg/doc/APIchanges: 2021-03-17 - f7db77bd87 - lavc 58.133.100 - codec.h Deprecated av_init_packet(). Once removed, sizeof(AVPacket) will no longer be a part of the public ABI. Refs #2302 <https://github.com/Genymobile/scrcpy/issues/2302>
2021-06-14 15:07:49 +08:00
}
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
for (;;) {
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
bool ok = sc_demuxer_recv_packet(demuxer, packet);
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
if (!ok) {
// end of stream
Properly report demuxer errors All demuxer errors were reported as "device disconnected", even if the failure was not related to device socket read.
2023-02-11 01:54:46 +08:00
eos = true;
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
break;
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
}
Do not merge config audio packets For video streams (at least H.264 and H.265), the config packet containing SPS/PPS must be prepended to the next packet (the following keyframe). For audio streams (at least OPUS), they must not be merged. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-19 00:37:58 +08:00
if (must_merge_config_packet) {
// Prepend any config packet to the next media packet
ok = sc_packet_merger_merge(&merger, packet);
if (!ok) {
av_packet_unref(packet);
break;
}
Extract packet merging Config packets must be prepended to the next media packet. Extract the logic to a new sc_packet_merger helper to simplify the demuxer code.
2023-02-11 05:52:40 +08:00
}
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
ok = sc_demuxer_push_packet(demuxer, packet);
Allocate AVPacket for local stream packet From FFmpeg/doc/APIchanges: 2021-03-17 - f7db77bd87 - lavc 58.133.100 - codec.h Deprecated av_init_packet(). Once removed, sizeof(AVPacket) will no longer be a part of the public ABI. Refs #2302 <https://github.com/Genymobile/scrcpy/issues/2302>
2021-06-14 15:07:49 +08:00
av_packet_unref(packet);
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
if (!ok) {
// cannot process packet (error already logged)
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
break;
}
}
Give a name to demuxer instances This will be useful in logs. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-19 07:13:54 +08:00
LOGD("Demuxer '%s': end of frames", demuxer->name);
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
Do not merge config audio packets For video streams (at least H.264 and H.265), the config packet containing SPS/PPS must be prepended to the next packet (the following keyframe). For audio streams (at least OPUS), they must not be merged. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-19 00:37:58 +08:00
if (must_merge_config_packet) {
sc_packet_merger_destroy(&merger);
}
Reduce latency by 1 frame To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder! On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording). On the client side, replace av_read_frame() by manual packetizing and parsing. <https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay> <https://trac.ffmpeg.org/ticket/3354>
2019-07-31 07:55:32 +08:00
Allocate AVPacket for local stream packet From FFmpeg/doc/APIchanges: 2021-03-17 - f7db77bd87 - lavc 58.133.100 - codec.h Deprecated av_init_packet(). Once removed, sizeof(AVPacket) will no longer be a part of the public ABI. Refs #2302 <https://github.com/Genymobile/scrcpy/issues/2302>
2021-06-14 15:07:49 +08:00
av_packet_free(&packet);
Make stream push packets to sinks Now that decoder and recorder implement the packet sink trait, make stream push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
finally_close_sinks:
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
sc_demuxer_close_sinks(demuxer);
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
end:
Properly report demuxer errors All demuxer errors were reported as "device disconnected", even if the failure was not related to device socket read.
2023-02-11 01:54:46 +08:00
demuxer->cbs->on_ended(demuxer, eos, demuxer->cbs_userdata);
Notify end-of-stream via callback Do not send a SDL event directly, to make stream independent of SDL.
2021-05-16 21:32:31 +08:00
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
return 0;
}
Update code style Limit source code to 80 chars, and declare functions return type and modifiers on a separate line. This allows to avoid very long lines, and all function names are aligned. (We do this on VLC, and I like it.)
2019-03-03 03:09:56 +08:00
void
Give a name to demuxer instances This will be useful in logs. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-19 07:13:54 +08:00
sc_demuxer_init(struct sc_demuxer *demuxer, const char *name, sc_socket socket,
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
const struct sc_demuxer_callbacks *cbs, void *cbs_userdata) {
Add an audio demuxer Add a demuxer which will read the stream from the audio socket. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-06 17:33:47 +08:00
assert(socket != SC_SOCKET_NONE);
Give a name to demuxer instances This will be useful in logs. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-19 07:13:54 +08:00
demuxer->name = name; // statically allocated
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
demuxer->socket = socket;
demuxer->sink_count = 0;
Notify end-of-stream via callback Do not send a SDL event directly, to make stream independent of SDL.
2021-05-16 21:32:31 +08:00
Properly report demuxer errors All demuxer errors were reported as "device disconnected", even if the failure was not related to device socket read.
2023-02-11 01:54:46 +08:00
assert(cbs && cbs->on_ended);
Notify end-of-stream via callback Do not send a SDL event directly, to make stream independent of SDL.
2021-05-16 21:32:31 +08:00
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
demuxer->cbs = cbs;
demuxer->cbs_userdata = cbs_userdata;
Make stream push packets to sinks Now that decoder and recorder implement the packet sink trait, make stream push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
}
void
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
sc_demuxer_add_sink(struct sc_demuxer *demuxer, struct sc_packet_sink *sink) {
assert(demuxer->sink_count < SC_DEMUXER_MAX_SINKS);
Make stream push packets to sinks Now that decoder and recorder implement the packet sink trait, make stream push packets to the sinks without depending on the concrete sink types.
2021-04-11 21:01:05 +08:00
assert(sink);
assert(sink->ops);
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
demuxer->sinks[demuxer->sink_count++] = sink;
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
}
Replace SDL types by C99 standard types Scrcpy is a C11 project. Use the C99 standard types instead of the SDL-specific types: SDL_bool -> bool SintXX -> intXX_t UintXX -> uintXX_t
2019-03-03 06:52:22 +08:00
bool
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
sc_demuxer_start(struct sc_demuxer *demuxer) {
Give a name to demuxer instances This will be useful in logs. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-19 07:13:54 +08:00
LOGD("Demuxer '%s': starting thread", demuxer->name);
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
bool ok = sc_thread_create(&demuxer->thread, run_demuxer, "scrcpy-demuxer",
demuxer);
Wrap SDL thread functions into scrcpy-specific API The goal is to expose a consistent API for system tools, and paves the way to make the "core" independant of SDL in the future.
2021-02-01 01:24:35 +08:00
if (!ok) {
Give a name to demuxer instances This will be useful in logs. PR #3757 <https://github.com/Genymobile/scrcpy/pull/3757>
2023-02-19 07:13:54 +08:00
LOGE("Demuxer '%s': could not start thread", demuxer->name);
Replace SDL types by C99 standard types Scrcpy is a C11 project. Use the C99 standard types instead of the SDL-specific types: SDL_bool -> bool SintXX -> intXX_t UintXX -> uintXX_t
2019-03-03 06:52:22 +08:00
return false;
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
}
Replace SDL types by C99 standard types Scrcpy is a C11 project. Use the C99 standard types instead of the SDL-specific types: SDL_bool -> bool SintXX -> intXX_t UintXX -> uintXX_t
2019-03-03 06:52:22 +08:00
return true;
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
}
Update code style Limit source code to 80 chars, and declare functions return type and modifiers on a separate line. This allows to avoid very long lines, and all function names are aligned. (We do this on VLC, and I like it.)
2019-03-03 03:09:56 +08:00
void
Rename stream to sc_demuxer For consistency with recorder and decoder, name the component which demuxes a "demuxer". And add the missing sc_ prefix.
2022-02-03 03:51:07 +08:00
sc_demuxer_join(struct sc_demuxer *demuxer) {
sc_thread_join(&demuxer->thread, NULL);
Add stream layer The decoder initially read from the socket, decoded the video and sent the decoded frames to the screen: +---------+ +----------+ socket ---> | decoder | ---> | screen | +---------+ +----------+ The design was simple, but the decoder had several responsabilities. Then we added the recording feature, so we added a recorder, which reused the packets received from the socket managed by the decoder: +----------+ ---> | screen | +---------+ / +----------+ socket ---> | decoder | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This lack of separation of concerns now have concrete implications: we could not (properly) disable the decoder/display to only record the video. Therefore, split the decoder to extract the stream: +----------+ +----------+ ---> | decoder | ---> | screen | +---------+ / +----------+ +----------+ socket ---> | stream | ---- +---------+ \ +----------+ ---> | recorder | +----------+ This will allow to record the stream without decoding the video.
2019-03-02 23:43:43 +08:00
}
Reference in a new issue Copy permalink