From 71fd238b0a8a888b51b89361bf0ad28b58d8e097 Mon Sep 17 00:00:00 2001 From: Romain Vimont Date: Thu, 7 Mar 2019 20:21:13 +0100 Subject: [PATCH] Update developer documentation for v1.8 --- DEVELOP.md | 79 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 54 insertions(+), 25 deletions(-) diff --git a/DEVELOP.md b/DEVELOP.md index 63d65b27..38c9c63e 100644 --- a/DEVELOP.md +++ b/DEVELOP.md @@ -32,7 +32,7 @@ The server is a Java application (with a [`public static void main(String... args)`][main] method), compiled against the Android framework, and executed as `shell` on the Android device. -[main]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/Server.java#L61 +[main]: https://github.com/Genymobile/scrcpy/blob/v1.8/server/src/main/java/com/genymobile/scrcpy/Server.java#L100 To run such a Java application, the classes must be [_dexed_][dex] (typically, to `classes.dex`). If `my.package.MainClass` is the main class, compiled to @@ -65,8 +65,8 @@ They can be called using reflection though. The communication with hidden components is provided by [_wrappers_ classes][wrappers] and [aidl]. [hidden]: https://stackoverflow.com/a/31908373/1987178 -[wrappers]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/wrappers -[aidl]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/aidl/android/view +[wrappers]: https://github.com/Genymobile/scrcpy/blob/v1.8/server/src/main/java/com/genymobile/scrcpy/wrappers +[aidl]: https://github.com/Genymobile/scrcpy/blob/v1.8/server/src/main/aidl/android/view ### Threading @@ -89,9 +89,9 @@ The video is encoded using the [`MediaCodec`] API. The codec takes its input from a [surface] associated to the display, and writes the resulting H.264 stream to the provided output stream (the socket connected to the client). -[`ScreenEncoder`]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java +[`ScreenEncoder`]: https://github.com/Genymobile/scrcpy/blob/v1.8/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java [`MediaCodec`]: https://developer.android.com/reference/android/media/MediaCodec.html -[surface]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L63-L64 +[surface]: https://github.com/Genymobile/scrcpy/blob/v1.8/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L69-L70 On device [rotation], the codec, surface and display are reinitialized, and a new video stream is produced. @@ -105,8 +105,9 @@ because it avoids to send unnecessary frames, but there are drawbacks: Both problems are [solved][repeat] by the flag [`KEY_REPEAT_PREVIOUS_FRAME_AFTER`][repeat-flag]. -[rotation]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L89-L92 -[repeat]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L125-L126 +[rotation]: https://github.com/Genymobile/scrcpy/blob/v1.8/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L90 +[repeat]: +https://github.com/Genymobile/scrcpy/blob/v1.8/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L147-L148 [repeat-flag]: https://developer.android.com/reference/android/media/MediaFormat.html#KEY_REPEAT_PREVIOUS_FRAME_AFTER @@ -124,11 +125,11 @@ All of them may need to inject input events to the system. To do so, they use the _hidden_ method [`InputManager.injectInputEvent`] (exposed by our [`InputManager` wrapper][inject-wrapper]). -[`EventController`]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/EventController.java#L70 +[`EventController`]: https://github.com/Genymobile/scrcpy/blob/v1.8/server/src/main/java/com/genymobile/scrcpy/EventController.java#L66 [`KeyEvent`]: https://developer.android.com/reference/android/view/KeyEvent.html [`MotionEvent`]: https://developer.android.com/reference/android/view/MotionEvent.html [`InputManager.injectInputEvent`]: https://android.googlesource.com/platform/frameworks/base/+/oreo-release/core/java/android/hardware/input/InputManager.java#857 -[inject-wrapper]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/wrappers/InputManager.java#L27 +[inject-wrapper]: https://github.com/Genymobile/scrcpy/blob/v1.8/server/src/main/java/com/genymobile/scrcpy/wrappers/InputManager.java#L27 @@ -153,7 +154,7 @@ Note that the client-server roles are expressed at the application level: - the server _serves_ video stream and handle requests from the client, - the client _controls_ the device through the server. -However, the roles are inverted at the network level: +However, the roles are reversed at the network level: - the client opens a server socket and listen on a port before starting the server, @@ -162,6 +163,9 @@ However, the roles are inverted at the network level: This role inversion guarantees that the connection will not fail due to race conditions, and avoids polling. +_(Note that over TCP/IP, the roles are not reversed, due to a bug in `adb +reverse`. See commit [1038bad] and [issue #5].)_ + Once the server is connected, it sends the device information (name and initial screen dimensions). Thus, the client may init the window and renderer, before the first frame is available. @@ -169,6 +173,8 @@ the first frame is available. To minimize startup time, SDL initialization is performed while listening for the connection from the server (see commit [90a46b4]). +[1038bad]: https://github.com/Genymobile/scrcpy/commit/1038bad3850f18717a048a4d5c0f8110e54ee172 +[issue #5]: https://github.com/Genymobile/scrcpy/issues/5 [90a46b4]: https://github.com/Genymobile/scrcpy/commit/90a46b4c45637d083e877020d85ade52a9a5fa8e @@ -177,17 +183,25 @@ the connection from the server (see commit [90a46b4]). The client uses 3 threads: - the **main** thread, executing the SDL event loop, - - the **decoder** thread, decoding video frames, + - the **stream** thread, receiving the video and used for decoding and + recording, - the **controller** thread, sending _control events_ to the server. +In addition, another thread can be started if necessary to handle APK +installation or file push requests (via drag&drop on the main window). -### Decoder -The [decoder] runs in a separate thread. It uses _libav_ to decode the H.264 -stream from the socket, and notifies the main thread when a new frame is -available. -There are two [frames] simultaneously in memory: +### Stream + +The video [stream] is received from the socket (connected to the server on the +device) in a separate thread. + +If a [decoder] is present (i.e. `--no-display` is not set), then it uses _libav_ +to decode the H.264 stream from the socket, and notifies the main thread when a +new frame is available. + +There are two [frames][video_buffer] simultaneously in memory: - the **decoding** frame, written by the decoder from the decoder thread, - the **rendering** frame, rendered in a texture from the main thread. @@ -195,9 +209,23 @@ When a new decoded frame is available, the decoder _swaps_ the decoding and rendering frame (with proper synchronization). Thus, it immediatly starts to decode a new frame while the main thread renders the last one. -[decoder]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/decoder.c -[frames]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/frames.h +If a [recorder] is present (i.e. `--record` is enabled), then its muxes the raw +H.264 packet to the output video file. +[stream]: https://github.com/Genymobile/scrcpy/blob/v1.8/app/src/stream.h +[decoder]: https://github.com/Genymobile/scrcpy/blob/v1.8/app/src/decoder.h +[video_buffer]: https://github.com/Genymobile/scrcpy/blob/v1.8/app/src/video_buffer.h +[recorder]: https://github.com/Genymobile/scrcpy/blob/v1.8/app/src/recorder.h + +``` + +----------+ +----------+ + ---> | decoder | ---> | screen | + +---------+ / +----------+ +----------+ + socket ---> | stream | ---- + +---------+ \ +----------+ + ---> | recorder | + +----------+ +``` ### Controller @@ -211,10 +239,10 @@ events_ to a blocking queue hold by the controller. On its own thread, the controller takes events from the queue, that it serializes and sends to the client. -[controller]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/controller.h -[controlevent]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/controlevent.h -[inputmanager]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/inputmanager.h -[convert]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/convert.h +[controller]: https://github.com/Genymobile/scrcpy/blob/v1.8/app/src/controller.h +[controlevent]: https://github.com/Genymobile/scrcpy/blob/v1.8/app/src/control_event.h +[inputmanager]: https://github.com/Genymobile/scrcpy/blob/v1.8/app/src/input_manager.h +[convert]: https://github.com/Genymobile/scrcpy/blob/v1.8/app/src/convert.h ### UI and event loop @@ -225,9 +253,10 @@ thread. Events are handled in the [event loop], which either updates the [screen] or delegates to the [input manager][inputmanager]. -[scrcpy]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/scrcpy.c -[event loop]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/scrcpy.c#L38 -[screen]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/screen.h +[scrcpy]: https://github.com/Genymobile/scrcpy/blob/v1.8/app/src/scrcpy.c +[event loop]: +https://github.com/Genymobile/scrcpy/blob/v1.8/app/src/scrcpy.c#L187 +[screen]: https://github.com/Genymobile/scrcpy/blob/v1.8/app/src/screen.h ## Hack