Hi guys. Please suggest for any improvements in my resumé.
Thanks in advance.
https://preview.redd.it/1oc0x7fx9yec1.jpg?width=1275&format=pjpg&auto=webp&s=951b4ecae16177719e55f5002ad8fd1e4103daca

I have been (slowly) working on a PoC maui app over the past year. One of the requirements of this app is to have a second window that's transparent. I looked all over the web but couldn't find a way to actually make the window transparent.
My solution was to create a native library that uses swift to make the window transparent. The repo for this library along with an example is located here. Hopefully this will help any others that have a similar requirement or that having trouble with transparent windows.
​
Here's a screenshot

Research is a big part of building, recently the core team has been diving deep into Open Transactions. In this thread, we'll explain what has been narrowed down so far about Open Transactions in (hopefully) simple terms!
https://preview.redd.it/jqqob3dwj0pb1.png?width=1200&format=png&auto=webp&s=c149f30a873ed68d0933b6f7bfdb21a314776a83
Open Transactions are a more efficient way for multiple parties to collaborate on creating a single transaction. Imagine working on a group project where everyone contributes their piece at different times. That's what Open Transactions can do for CKB users.
​
https://preview.redd.it/c57z2d4zj0pb1.png?width=1200&format=png&auto=webp&s=1029be8af586a69ba827b9729657c2855abd8c10
Open Transactions help users and services communicate and share information more easily. They're especially useful when multiple people or services need to work together on a single transaction.
​
https://preview.redd.it/19b9hb52k0pb1.png?width=1200&format=png&auto=webp&s=fc6a13a536cebcbe21f494eb932bbd8657a644b1
This innovative approach helps in two main ways: (1) allowing participants to add their part one after the other, and (2) letting third parties combine different transactions into one. It's like everyone working together on a puzzle!
​
https://preview.redd.it/4386b3b8k0pb1.png?width=1200&format=png&auto=webp&s=e3f17fe376e7814abb31321fd213a27dc956fd88
But what does it mean for decentralized applications (dApps) on CKB? Open Transactions can have a significant impact on #dApps in various ways, leading to better user experiences and more powerful applications.
First, they enhance composability, making it easier for dApps to interact and compose with other dApps, services, or users. This enables the creation of more complex and feature-rich applications that can leverage multiple services and components.
https://preview.redd.it/7o19fhoak0pb1.png?width=1200&format=png&auto=webp&s=4c9d4518d8b595e60770f9d09568a3f6ae246bb5
Improved transaction efficiency is another benefit. Incremental transaction construction and aggregation facilitated by Open Transactions can help optimize network usage, reducing congestion and improving the overall performance of dApps.
​
https://preview.redd.it/652man3dk0pb1.png?width=1200&format=png&auto=webp&s=ffe70129e7b063db951b5b8afbc18296df00b646
Greater flexibility and choice for users are possible with Open Transactions. Users can choose different services at each step of the transaction process, giving them more control and options when using dApps and interacting with the CKB ecosystem.
Enhanced privacy is another advantage. Open Transactions can enable privacy-preserving techniques, like CoinJoin, to help anonymize transactions and protect user privacy. This can make dApps on CKB more appealing to users who value privacy and security.
​
https://preview.redd.it/wv2lqvggk0pb1.png?width=1200&format=png&auto=webp&s=32bab4da68005fe042ae98bd867d37f7c264faba
Moreover, Open Transactions help mitigate the hot cell problem, where multiple transactions compete for the same input cell. This reduces the need for users to repeatedly create and sign transactions, resulting in a smoother experience when using dApps.
In conclusion, Open Transactions on CKB simplify collaboration and communication between different parties involved in creating a transaction. They provide a more efficient and secure way to complete transactions on the network.
​
https://preview.redd.it/eznwbrklk0pb1.png?width=1200&format=png&auto=webp&s=265f6d78cb956b7ea9563eb45cb1af123e218bb8
With their potential to enhance the functionality, efficiency & user experience of dApps on CKB, Open Transactions can contribute to the growth & success of the CKB ecosystem. If you want to dive into the deep end, check out this post on Nervos Talk:
https://talk.nervos.org/t/exploring-the-ckb-otx-paradigm-accomplishments-and-insights-from-building-a-transaction-streaming-prototype/7346

1 Synopsis

The CKB Open Transaction (OTX) allows users to create transactions that are open to change. Users send these open transactions to a network, where many autonomous agents provide services for assembling CKB transactions from open transactions.
This report will recap the accomplishments and insights we gained in designing a OTX proposal and building a prototype. The purpose of this prototype is to test a possible design, understand its pros and cons, and find unexpected obstacles and potential improvements to the core protocol. Most of the OTX prototyping work is done by Ethan Yuan and myself. Credits go to Jiandong for helping us design and implement the first version of OTX Lock based on the Instructions List.
The OTX Prototype project can be divided into four areas, each of which will be discussed in a separate chapter later in the text. Please note that all the work here is for prototype purposes, the OTX protocol could be very different from this prototype in future iterations. Here is a summary of these areas:

• OTX Format: An extensible transaction format used to describe a CKB transaction along with its attached metadata.
• OTX Streaming Pattern: A pattern that involves placing open transactions in a stream and dividing the process of constructing transactions into smaller, reusable components.
• OTX Lock: A partial signing lock script that excludes certain transaction properties from the signature, allowing the signature to remain valid even if those properties are altered.
• OTX Framework: A framework for designing and developing dApps utilizing OTX Lock, OTX Streaming Pattern, and OTX format.

2 OTX Format

Constructing a CKB transaction can be complex. Even a basic CKB transfer transaction requires gathering live cells, forming an output for the recipient, allocating sufficient fees, and creating the change output for the remaining CKB.
To address the issue of transaction construction, several teams have formulated step-by-step approaches and developed proprietary formats to store intermediate results. For instance, ckb-cli incorporates additional fields for signing informations 2 and stores them into a JSON file within the tx sub-command. Similarly, Lumos offers the TransactionSkeleton interface to construct transactions. However, the use of proprietary formats prevents collaboration across different tools. For example, it’s impossible to construct a transaction in Lumos first, then sign it using ckb-cli.
To address this, the OTX Format has been introduced as a transaction format specification standard that promotes collaboration and component reuse, enabling seamless movement of transactions between components, processes, and machines across the network. The proposed specification, submitted as PR#406 4, is available for review in the Nervos Network/rfcs repository.
The OTX Format stores metadata as key-value pairs, with keys as 32-bit integers and values as opaque data types. The key determines the encoding of the corresponding value. Applications serialize and deserialize values of interest while treating others as raw byte arrays. This modular design decouples dependencies and enables the creation of simple, reusable, and composable components.
To avoid key conflicts, the community should collectively determine the allocation of keys via the RFC process, where the implication and usage of the keys are defined, specified, and reserved in the RFC repository. The RFC repository serves as a registry for sharing information among transaction construction applications. Proprietary keys can be used by applications without community consensus if they are not transmitted over the public network. A rich library of shared keys is essential for facilitating successful collaboration among applications using the OTX Format.
A typical pattern to use the keys is to store commands that instruct a service to construct transactions. For instance, a service may have a key named TRANSFER_CKB . The code in Listing 1 below shows an example of the value. From the code snippet, it is easy to derive that the user “alice@ckb” intends to transfer 1,000 CKB to “bob@ckb”, where the fee rate must be within the range of 0.0001 to 0.00011. As depicted in the example, the sender and recipient addresses use an email-like format, and the service can lookup the corresponding CKB lock script in the name registries. Clients who wish to transfer CKB can initiate an open transaction by setting TRANSFER_CKB without needing to know the underlying details. The service will use the TRANSFER_CKB key to generate a valid CKB transfer transaction, much like calling a method transferCKB . This pattern has evolved into the OTX Streaming Pattern, which is discussed in Chapter 3.
Listing 1: Example value of the key TRANSFER_CKB

{ "from": "alice@ckb", "to": "bob@ckb", "amount": "1000", "minFeeRate": "0.0001", "maxFeeRate": "0.00011", } 

3 OTX Streaming Pattern

The Streaming Pattern is an architecture that enables software to react and operate as events occur. This pattern allows for software components to work together in a real-time, decoupled, and scalable fashion. It is well suited for the development of modern real-time distributed systems, such as dApps. Confluent offers an excellent introduction to the Streaming Pattern; 1 and ReactiveX an example of the Streaming Pattern framework that supports numerous languages.
The OTX Streaming Pattern uses the OTX Format as the event payload. Autonomous OTX Agents subscribe to the OTX Stream to receive notifications of new open transactions and then process them selectively based on their criteria. The agents also emit modified or new transactions, which are merged into the OTX Stream.
Figure 1: OTX Streaming Pattern
📷
As demonstrated in Figure 1, one of the components is optional. For instance, an agent that receives transactions from RPC will only require an OTX Source, while an agent that sends finished open transactions to a CKB node only needs an OTX Processor.
To further illustrate the pattern, here are some examples of agents.

1. RPC Emitter receives transactions from an RPC endpoint and forwards them to the OTX Stream.
2. CKB Sender sends completed transactions to the CKB RPC endpoint.
3. Signer identifies signing requests in the stream via a registered OTX Format key and stores them locally. A wallet application retrieves the pending signing requests from the Signer agent for users. These requests are presented to users as a list in the application’s UI. After reviewing the requests, users authorize the wallet application to sign the transactions using their private keys. The application then submits the signed transactions to the RPC Emitter agent. Once the Signer detects the presence of the signed transactions in the stream, it removes the corresponding local unsigned versions.
4. Atomic Swap Matcher functions as an order book for the atomic swap proposals. It indexes the swap proposals locally and offers an RPC to search for these proposals. Clients have the option to search their own proposals to check their status or search proposals made by others to take the orders. The agent also tries to merge matched proposals and emit the merged open transaction to the OTX Stream. Therefore, the Atomic Swap Matcher acts the roles of both OTX Processor and OTX Source.

4 OTX Lock

CKB lock scripts typically require users to sign the entire transaction, which involves two steps to complete an open transaction. First, users need to send the initial open transaction. Then, they must wait for its completion before signing it. In case of failure, users have to wait for a new completed transaction to sign again.
The OTX Lock offers partial signing mechanisms that protect specific properties of a transaction while leaving the other parts free to change. The signature remains valid as long as the signed properties remain unchanged. This enables users to pre-sign open transactions using OTX Lock, facilitating a fire-and-forget style similar to typical CKB transaction processes.
In the following sections, we will discuss two different ways to design the Lock: the Instruction List Lock and the Sighash Lock. Furthermore, Chapter 6 will present several new directions for further exploration.

4.1 Instructions List Lock

The Instructions List Lock enables users to specify the content for inclusion in the digest message for signing through an instructions list. Each instruction adds transaction properties to the digest, such as a full input cell or the number of output cells. We previously presented a design proposal in the RFC: Composable Open Transaction Lock Script 2.
We have tried to use the lock in different scenarios; two of the most studied ones are Atomic Swap and Unilateral Payment.

• In Atomic Swap, users send swap proposals as open transactions. These proposals describe the assets and quantities users wish to obtain, as well as the assets and quantities they want to pay. Each proposal is unbalanced, and it is the responsibility of the Atomic Swap OTX Agent to merge the matched proposals into a balanced CKB transaction.
• In Unilateral Payment, we want to design a mechanism based on open transactions to pay assets without interactions from payees. The requirement is from the scenario that users pay small amounts of CKB or user-defined tokens (UDT) but don’t want to incur the storage costs of creating a cell for the payee.

The design of the transaction structures and the instruction lists can be found in an older revision of the repository EthanYuan/open-transaction-pool.
During the experiments to design those scenarios, we experienced significant drawbacks of the proposal. It’s challenging to create a secure instructions list, and the field-by-field inclusion algorithm is not powerful enough for most complex dApps.
The security issues originate from the uncertainty of uncovered transaction fields, making it difficult to predict how open transactions will be consumed. In subsequent paragraphs, we will recapitulate two categories of security issues.
The first category is the Replay Attack, which occurs when an attacker reuses the signature created from a specific instruction list to gain access to other assets owned by the same user. This can occur when the instructions list does not include a field unique to the current transaction. To prevent this, users must avoid signing any unknown digest messages. Additionally, instructions lists must include at least one input cell outpoint to ensure uniqueness.
The second category of security issues arises from the absence of support for Cell Grouping. One of the key features of OTX is the ability to merge open transactions. However, when multiple transactions are merged, it becomes impossible to reference cells based on their absolute locations. To address this, the proposal introduces the method of referencing a cell using the relative index, which is a number relative to a base value. While anyone is allowed to modify the base values, it is typically the responsibility of the Agent who merges open transactions to set the appropriate values without compromising the existing signatures. See how Listing 3 sets the base values for the Open Transaction 2 after merging two atomic swap proposals.
Listing 3: Atomic Swap Merging Example

Open Transaction 1: # Alice wants to get 20 SUDT X by paying 200 CKB inputs: 0: CKB 200 owned by Alice outputs: 0: SUDT X 20 owned by Alice inputWitnesses: 0: # Initially, both base values are set to 0 inputBase: 0 outputBase: 0 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..." Open Transaction 2: # Bob wants to get 200 CKB by paying 20 SUDT X inputs: 0: SUDT X 20 owned by Bob outputs: 0: CKB 200 owned by Bob inputWitnesses: 0: # Initially, both base values are set to 0 inputBase: 0 outputBase: 0 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..." Open Transaction 1 + 2: # Make proposals from Alice and Bob inputs: 0: CKB 200 owned by Alice 1: SUDT X 20 owned by Bob outputs: 0: SUDT X 20 owned by Alice 1: CKB 200 owned by Bob inputWitnesses: 0: # Transaction 1 comes first, so the base values are zeros. inputBase: 0 outputBase: 0 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..." 1: # Shift the base values by one to reference the correct cells. inputBase: 1 outputBase: 1 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..." 

Allowing anyone to set the base values is dangerous, as adversaries can exploit this by reusing a cell in different open transactions. This is possible because output cells in the CKB transaction lack unique identifications. Let’s revise the swap example by splitting Bob’s proposal into two identical proposals as in Listing 4. By reusing the last output, Bob only receives 100 CKB instead of the intended 200 CKB.
Listing 4: Cell Output Reusing

Open Transaction 1: # Alice wants to get 20 SUDT X by paying 200 CKB inputs: 0: CKB 200 owned by Alice outputs: 0: SUDT X 20 owned by Alice inputWitnesses: 0: inputBase: 0 outputBase: 0 instructions: - "inputs[inputBase + 0]" - "outputs[outputBase + 0]" signature: "0x..." Open Transaction 2: # Bob wants to get 100 CKB by paying 10 SUDT X inputs: 0: SUDT X 10 owned by Bob outputs: 0: CKB 100 owned by Bob inputWitnesses: 0: inputBase: 0 outputBase: 0 instructions: - "inputs[inputBase + 0]" - "outputs[outputBase + 0]" signature: "0x..." Open Transaction 3: inputs: 0: SUDT X 10 owned by Bob outputs: # This output is identical to the one in Transaction 2 0: CKB 100 owned by Bob inputWitnesses: 0: inputBase: 0 outputBase: 0 instructions: - "inputs[inputBase + 0]" - "outputs[outputBase + 0]" signature: "0x..." Open Transaction 1 + 2 + 3: # Make proposals from Alice and Bob inputs: 0: CKB 200 owned by Alice 1: SUDT X 10 owned by Bob 2: SUDT X 10 owned by Bob outputs: 0: SUDT X 20 owned by Alice 1: CKB 100 owned by Bob # The adversary removes the output from Open Transaction 3 inputWitnesses: 0: inputBase: 0 outputBase: 0 instructions: - "inputs[inputBase + 0]" - "outputs[outputBase + 0]" signature: "0x..." 1: inputBase: 1 outputBase: 1 instructions: - "inputs[inputBase + 0]" - "outputs[outputBase + 0]" signature: "0x..." 2: inputBase: 2 # The adversary sets this to 1 to reuse the output in Transaction 2 outputBase: 1 instructions: - "inputs[inputBase + 0]" - "outputs[outputBase + 0]" signature: "0x..." 

In conclusion, a cells-grouping mechanism that prevents tampering by adversaries is crucial for merging open transactions. Implementing the mechanism in CKB is much simpler compared to using contracts, but it requires careful design to minimize overhead and ensure compatibility.
There is a concept called script group, which is similar to cell grouping. It involves grouping cells based on scripts and arguments and then running the script once for each group. However cells in an open transaction often do not align with a script group. What’s even worse is that there are often overlaps between open transactions and script groups. If a lock script fails to iterate through all the cells in the group or limit the number of cells, it becomes vulnerable to an attack where adversaries can append input cells that belong to the same script group. These additional inputs get unlocked along with existing ones without requiring a new signature from the user. This clearly violates the intentions of the user. Listing 5 depicts an OTX Lock implementation that is vulnerable, as it only verifies the first input witness in the group. Listing 6 demonstrates an attack that exploits this vulnerability. Pay attention to input 2 of the merged transaction. Specifying the number of input cells when signing can solve this issue, but would prevent merging open transactions from the same user into a single CKB transaction. This is a common scenario in merchant apps where a buyer purchases multiple items in a single transaction. Restricting the number of input cells would result in a complex design where goods belonging to the same seller cannot be combined in the same transaction.
Listing 5: A vulnerable OTX Lock which verifies only the first input witness in the group

function main() { const { instructions, signature } = getInputWitnessInGroup(0); const digest = generateDigest(instructions); verifySignature(digest, getPubkeyFromArgs(), signature); } 

Listing 6: An attack that appends cells in the same script group

Open Transaction 1: # Alice wants to get 20 SUDT X by paying 200 CKB inputs: 0: CKB 200 owned by Alice outputs: 0: SUDT X 20 owned by Alice inputWitnesses: 0: # Initially, both base values are set to 0 inputBase: 0 outputBase: 0 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..." Open Transaction 2: # Bob wants to get 200 CKB by paying 20 SUDT X inputs: 0: SUDT X 20 owned by Bob outputs: 0: CKB 200 owned by Bob inputWitnesses: 0: # Initially, both base values are set to 0 inputBase: 0 outputBase: 0 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..." Open Transaction 1 + 2: # Make proposals from Alice and Bob inputs: 0: CKB 200 owned by Alice 1: SUDT X 20 owned by Bob # Adversaries append the cell which is in the same script group as input 0 2: CKB 1000 owned by Alice outputs: 0: SUDT X 20 owned by Alice 1: CKB 200 owned by Bob inputWitnesses: 0: # Transaction 1 comes first, so the base values are zeros. inputBase: 0 outputBase: 0 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..." 1: # Shift the base values by one to reference the correct cells. inputBase: 1 outputBase: 1 instructions: # The first input starting from inputBase - "inputs[inputBase + 0]" # The first output starting from outputBase - "outputs[outputBase + 0]" signature: "0x..." 

Another drawback that was mentioned earlier is that the Instruction List Lock is not powerful enough. For instance, it is not possible to replicate the logic of Anyone-Can-Pay (ACP) Lock using an instruction list, because there are no commands available for performing arithmetic calculations and comparisons. We have been cautious about adding new instructions, due to the possibility of requiring endless additional instructions. The sustainable approach would be through script composition, where dApps expose verified assertions for signing. There are two threads 4 in the CKB GitHub repository relevant to this topic.

4.2 Sighash Lock

To simplify the design of the instructions list, why not limit users to a set of established patterns for signing transactions? This is where the concept of OTX Sighash Lock comes into play.
The OTX Sighash Lock design is modeled on the Bitcoin Sighash pattern. If you want to learn more about the Bitcoin Sighash, check out the tutorial from saylor.org: CS120: Bitcoin for Developers I, Elliptic Curve Signatures. For those interested in implementing OTX Sighash Lock, refer to the following Github repository: EthanYuan/otx-sighash-lock.
Although the Sighash pattern provides only six options, it does not simplify the issue. Rather, the complexity is shifted elsewhere. Sighash pattern requires an elaborate transaction layout design, as seen in the Atomic Swap demo, where users need to prepare a dedicated cell for the open transaction instead of using existing ones. For a reference, see the atomic swap documentation 2. By contrast, the older Instructions List Lock design gave users more freedom to choose from existing cells.

5 OTX Framework

OTX Framework combines OTX Format, OTX Streaming Pattern, and OTX Lock together to provide an easy-to-use instance for developers.
The central component of the framework is the implementation of the OTX Streaming Pattern, referred to as the open transaction pool. A Rust Proof of Concept (PoC) is available on GitHub at EthanYuan/open-transaction-pool 1.
Using the open transaction pool as a foundation, we can develop various agents, such as Atomic Swap and Signer. In our vision, there will be a public marketplace where agents can be shared. Developers can easily incorporate agents from this marketplace and construct complex transactions through agent composition. Certain agents, such as the one designed to collect live cells based on a specific criterion, can significantly facilitate the transaction construction process.
The SDK acts as the interface of the framework for developers. We already have code snippets to work with Open Transactions in Rust and Javascript 2, but these are far from an SDK for the framework.

6 Future Works

6.1 New Directions of OTX Lock

Both the Instruction List Lock and Sighash Lock require a redesign of the Cell Grouping mechanism. In the short term, it is necessary to integrate a solution into the OTX Lock. In the long term, further research is needed to determine how to support cell grouping in the CKB transaction structure.
There are also other mechanisms of partial signing, such as signing the user intent rather than specific fields. An intent is a message indicating the operation that the user wants to perform, such as the example in Listing 7.
Listing 7: An intent to swap 10 CKB with 50 SUDT

{ "app": "0x...", "nonce": 1, "command": "swap", "from": { "assets": "ckb", "balance": "10", }, "to": { "assets": "sudt", "id": "0x...", "balance": "50", } } 

The dApp checks the intent has been successfully carried out. The transaction properties that do not affect the intent execution are free to change. For example, the intent above does not care which inputs have been collected to provide the 10 CKB balance; it only needs to check whether the user’s CKB balance has decreased by exactly 10 CKB.
The dApp verifies the successful execution of intent. The transaction properties that do not affect execution are free to alter. For instance, the specific inputs used to attain the 10 CKB balance do not matter - only the user’s CKB balance being reduced by precisely 10 CKB matters.
To prevent repay attacks, it’s crucial to implement a mechanism to make the intent unique, such as the nonce field in the example.
Intent functions as the instruction for constructing a transaction as well. This means that dApps can use the same logic for both constructing and verifying transactions. The verification code rebuilds the transaction using the intent and checks that it matches the target transaction being verified.

6.2 Others

• A public P2P network to exchange open transactions is crucial for the adoption of dApps. We could incorporate an opt-in protocol in CKB to relay open transactions.
• Create a uniform interface for transaction construction across SDKs, for both CKB and open transactions. Developers can experience a seamless transition with a turn to the open transactions solution.
• Improve tools for better developer experience.
  • We already have code snippets to work with Open Transactions in Rust and Javascript 2. We could improve the JavaScript SDK further. Besides working with the OTX Format, the SDK must hide the complexity of the OTX Lock and support a mechanism to seamlessly integrate the features provided by agents.
  • We have a Rust PoC for the Streaming Pattern available on GitHub at EthanYuan/open-transaction-pool 1. We could implement the Streaming Pattern and all agents in JavaScript.

​
We're excited to see the potential of Open Transactions unfold and how they can revolutionize the way people interact with the CKB network and its dApps. Keep an eye out for more updates and developments from Nervos!
​
https://preview.redd.it/haovtlocl0pb1.png?width=1200&format=png&auto=webp&s=65645b3b5480228c4205a4650d3b20e14707fa37

01 Introduction

This is the third article in Goby's community memory shellcode series. The first article, "Ghost King in Shell - JAVAWEB Memory Shellcode [Cognitive]" introduced the history and classification of JavaWeb memory shellcode technology, and introduced common JavaWeb memory shellcode technology from a cognitive perspective; the second article, "Using Goby to Inject Memory Shellcode with Deserialization Vulnerabilities [Exploit]" mainly introduced how to combine memory shellcode with vulnerabilities to enable Goby to inject memory shellcode with one-click through deserialization vulnerabilities, and integrate with Goby's PoC and extension system. Users only need to click a few buttons to complete the injection of vulnerabilities with one-click.
This article mainly introduces some technical details used in the process of using Goby to inject memory shellcode with one-click through deserialization vulnerabilities, based on the first two articles. Of course, users do not need to know these details during the injection process using Goby PoC, but understanding and learning the technology helps to grasp some common ideas.
This article is mainly divided into three parts: "Exploiting Pre-Vulnerabilities", "Generating Memory Shellcode", and "Using Memory Shellcode", sharing some technical points and details or pitfalls related to Goby, and welcome everyone to discuss together.
Here is a brief demonstration of the use of some related technologies. The following video demonstrates the use of Goby to inject a Filter-type memory shellcode with one-click through deserialization, and carry false information through a custom URLClassLoader to avoid security personnel's investigation. The purpose is achieved by clearing the log without a trace.
​
https://reddit.com/link/126hgcu/video/khixam196uqa1/player
​
> The one-click Memory shellcode injection feature of Goby can be used for free in the community version.
> [Get version](https://gobies.org)
​

02 Pre-vulnerability Exploitation

First, let's talk about the pre-vulnerability exploitation. As mentioned in previous articles, from the perspective of practical vulnerability exploitation and weaponized development, we tend to inject a memory shellcode with one click during the vulnerability exploitation process, rather than obtaining a JSP webshell first and then converting it into a memory shellcode. Therefore, here we need to consider how to directly execute the implantation action of the memory shellcode during the vulnerability exploitation process.
2.1 Dynamic Loading and Class Initialization
In most current vulnerability exploits, if you want to execute complex malicious attack logic, you usually use a new URLClassLoader, the current thread's class loader, or a custom class loader to load and initialize malicious class bytecode. In different exploitation scenarios, you can choose different class loaders according to the situation, but sometimes you cannot choose and need to adjust according to the situation:

• Use a new URLClassLoader. If not specified, the system class loader is used as the parent ClassLoader by default, which is the AppClassLoader.
• Use the context class loader of the current thread, generally obtained using `Thread.currentThread().getContextClassLoader()`.
• Create a custom class loader, generally by defining a method for loading classes through bytecode, which is like encapsulating a public `defineClass` method.
• In some exploitation scenarios, it is not possible to customize ClassLoader, such as using BCEL ClassLoader for exploitation.

When using different ClassLoaders to load malicious classes in different situations, different problems will be faced:

• When using the context class loader of the current thread or cannot control the class loader, there may be a situation where the same class name cannot be loaded twice and additional processing is required.
• When using special ClassLoaders such as BCEL ClassLoader, due to the problem of loading across classes, some classes and interfaces need to be accessed and called through pure reflection, which requires a relatively large amount of physical work.

When dynamically loading classes during vulnerability exploitation, it is generally necessary to manually break the parent delegation mechanism and inject the malicious class into the system. Class initialization is closely related to class loading. Usually, in malicious code, some initialization malicious logic will be written, which can generally be written in the static statement block or public parameterless constructor:

• The static statement block is executed once when the class is loaded and only executed once during its lifecycle.
• The public parameterless constructor is called during class initialization, and it is called each time a new class instance is created.

Therefore, you can choose a class loader according to the specific situation and place the malicious logic in an appropriate location.
2.2 Echo and Memory Shellcode
After the Goby deserialization implantation extension went online, I enhanced and corrected the exploitation of deserialization vulnerabilities in the vulnerability library. Friends familiar with Goby may know that Goby's detection of vulnerability exploitation is divided into PoC and EXP. When facing native Java deserialization, the original detection and exploitation procedures were:

• PoC uses URLDNS combined with Goby's built-in dnslog platform GodServer for vulnerability detection.
• EXP uses the bytecode of YSOSERIAL, dynamically replaces the hex value of the command execution part, and writes the command execution.

The above logic is used to detect vulnerabilities, which is the way most people detect deserialization vulnerabilities. Technically, there is no problem with this detection method, but in practice, the following problems may be encountered:

1. Due to unstable network or DNSLOG platform, it may not be possible to receive DNSLOG or DNSLOG may have a long delay.
2. Vulnerability exploitation only performs command execution, and it is often impossible to determine whether the vulnerability exploitation is successful or what the result of the vulnerability execution is.
3. In a scenario where there is no outbound network connectivity, it is not possible to perform a reverse shell or execute more advanced actions. In terms of practicality for real-world scenarios, its usability is quite poor.

Therefore, to address the usability issues in practical scenarios, all subsequent updates to the vulnerability exploitation PoCs have adopted echo-based techniques to return the command execution results in the response. As for the exploits (EXP), they are directly injected into the memory as a shellcode, saving a lot of intermediate processes.
https://preview.redd.it/om35mjpa8uqa1.png?width=3526&format=png&auto=webp&s=50d20516faf1de88f808f0fd6d293214a179cd96
2.3 In constructing an echo
It involves locating the critical request, searching memory, and other technical points. And to inject a memory shellcode, it is necessary to prepare a highly available memory shellcode for the vulnerability environment. With these technical supports, the problems mentioned above can be solved without the need for third-party dnslog, OOB, etc., directly conducting high-precision detection and utilization of vulnerabilities.
There are many types of vulnerabilities, and there are also many types that can provide arbitrary code execution, such as Java native deserialization vulnerabilities, Fastjson/Jackson/XStream deserialization vulnerabilities, SpEL/Ognl expression injection, etc. However, many situations require additional utilization methods to complete the vulnerability utilization process. Taking advantage of the native deserialization as an example, some modifications of the utilization chain are listed to directly inject memory shellcode.

• The Transformer[] utilization chain is the most classic utilization chain, generally chain a Runtime.getRuntime().exec() or new ProcessBuilder().start() to execute commands. If you want to execute additional functions, you can also use new URLClassLoader().loadClass() to perform remote class loading. Without going online, you can write JS to inject malicious classes by using com.sun.org.apache.bcel.internal.util.ClassLoader.loadClass(), org.mozilla.javascript.DefiningClassLoader().defineClass(), new ScriptEngineManager().getEngineByName("JavaScript").eval() methods, and one-click utilization of memory shellcode.

https://preview.redd.it/nqcjxw0s8uqa1.png?width=2948&format=png&auto=webp&s=b39a2f0e380f3bdb64bbd2170d01c25b57882f12

• BeanShell chain, although Bsh supports all Java syntax and many loose writing methods, is ultimately a script language parser. If these writing methods are used or arrays are used in the script, related implementation classes' methods may be called during the deserialization process, and Interpreter objects may be used, which could result in a NullPointerException. Therefore, it is still possible to use ScriptEngineManager to parse JS and execute the memory shellcode.

https://preview.redd.it/6q0igqey8uqa1.png?width=2948&format=png&auto=webp&s=8301ebe53d4212e9c5f9a7b83be5ac15df10a1fa

• In the original version, `C3P0` chain used PoolBackedDataSource for remote class loading to exploit vulnerabilities. However, C3P0 can also use Tomcat's getObjectInstance method to call the eval method of ELProcessor for expression injection. This allows injection of memory shellcode through EL expressions, and can also be achieved through other methods such as Groovy, SnakeYaml, etc.

https://preview.redd.it/983m1ie29uqa1.png?width=2200&format=png&auto=webp&s=a940d51c4f83e25f1aa5ac09dcee303a426d9967
Here are several techniques that link the deserialization exploit chain to memory shell. There are also many other exploit situations that can be “saved by the bell”. Considering the length of the article, further elaboration on these techniques will not be discussed here.

03 Generating In-Memory Shellcode

After discussing the direction of vulnerability exploitation, we will now discuss some technical details involved in generating in-memory shellcode.
3.1 Dynamic Code Generation Techniques
Considering different vulnerability exploitation points, different exploitation scenarios and requirements, and different personnel's habits and preferences, the content of in-memory shellcode cannot be fixed in practical environments and needs to be dynamically generated based on various configurations.
Therefore, we use javassist to dynamically generate and write malicious bytecode of in-memory shellcode. In the process of preparing in-memory shellcode, we will face some requirements:

• The exploitation method of the vulnerability is fixed, such as command execution, commonly used tools such as Behinder, Godzilla, or self-developed webshell interaction tools, and most of them are reusable custom vulnerability exploitation methods;
• In-memory shellcode can customize URL and password, in addition to the common AES key, additional authentication mechanisms can also be set;
• Any in-memory shellcode technique can be freely selected, and any exploitation method can be used to quickly generate dynamically.

Therefore, I finally abstract the key logic into a same method, whose first two parameters are Request and Response objects. No matter it is command execution, Behinder, Godzilla, etc., their own logic can be injected into this method. For different middleware, due to different encapsulation and implementation, extra judgment and processing are performed before entering the key logic to make the final processing logic consistent.
For example, below is the core logic of Behinder:
https://preview.redd.it/wkwgl98c9uqa1.png?width=2550&format=png&auto=webp&s=8628569046170ba7c82fb2b32312dc4de63083fe
Here is the core logic of Godzilla:
https://preview.redd.it/ee6idwph9uqa1.png?width=2656&format=png&auto=webp&s=2422c46fe77be58d980a2ecc9b6b47ef6c84ed88
Here is the logic of command execution:
https://preview.redd.it/pxyqf7gk9uqa1.png?width=2632&format=png&auto=webp&s=be1eb3f2c94be2d7ea2c11cf152d66c47637b325
After determining the parameters to be used, bytecode can be assembled based on different Memory shellcode types and exploitation methods, with critical methods inserted into malicious classes in sequence, ultimately forming a complete memory shell.
3.2 ClassLoader Issues
As mentioned before, when dynamically loading and initializing a malicious class, it is important to consider the ClassLoader selection. This remains true after the Memory shellcode is loaded, as ClassLoader issues still need to be carefully considered.
In the first case, as the Memory shellcode file itself, the instance should generally be placed in a key position for processing routes, such as in a Map member variable of the global context. In this case, it is necessary to pass a reference to an instance, and register an instance of the shell's own object in a critical position within the system during malicious class initialization.
However, there are exceptions, such as in the Struts2 framework, where the key position stores the class name rather than the class instance. When processing routes, if a mapping is found, the class instance is dynamically created and its execute method is called for processing. Therefore, when injecting a malicious memory shell, the class name and route mapping should not be the only considerations, as the memory shell's own class should also be loaded into the critical context, allowing it to find our injected malicious class during class instantiation.
In terms of exploitation methods, in addition to command execution and feedback, the key logic of a Memory shellcode is still achieved through the transmission of class bytecode. In addition to the previously mentioned URLClassLoader, custom ClassLoader, and thread context ClassLoader, there are still many tricks that can be used, such as:

• Registering a class using java.lang.reflect.Proxy#defineClass0()
• Registering a class directly in the JVM using sun.misc.Unsafe#defineAnonymousClass()
• Using some wrapper classes that may call some uncommon ClassLoaders, such as jdk.nashorn.internal.runtime.ScriptLoader#installClass() and com.sun.naming.internal.VersionHelper#loadClass()

In addition to the above, JavaSec group members have shared some other methods:

• jxxload_help.PathVFSJavaLoader#loadClassFromBytes
• org.python.core.BytecodeLoader1#loadClassFromBytes
• sun.org.mozilla.javascript.internal.DefiningClassLoader#defineClass
• java.security.SecureClassLoader#defineClass
• org.mozilla.classfile.DefiningClassLoader#defineClass

3.3 Exploitation Methods
For Memory shellcode exploitation methods, the three most common types are command execution and feedback, and the Behinder and Godzilla shells, each with their own advantages:

• Command execution and feedback: Simple command execution with feedback visible.
• Behinder and Godzilla shells: Both provide advanced features that can be selected as needed.

In addition to the typical web shell exploitation methods, the latest trend is the infiltration of tunneling shells. After obtaining a web shell, attackers typically use this machine as a jump point for further intranet penetration. This requires a clear tunneling flow.
Previously, the common approach was to upload a traffic forwarding tool such as FRP to the target server and use this tool for traffic forwarding. If the network layer is not fully port mapped, this can also involve port reuse and other techniques.
However, with a memory shell, a tunneling shell can be easily created with one click, and the appropriate client can be used for direct connection, achieving a true "one-stop" solution.
https://preview.redd.it/yees98yz9uqa1.png?width=1432&format=png&auto=webp&s=628b9901b2d139d5320139152274062f491ed44a
3.4 Agent No File
The AgentNoFile technology implemented by Master rebeyond provides us with the ability to directly call the JVMTI interface without the need to provide Agent.jar or Agent.so. With this capability, we can inject Agent-type memory shellcode without file landing.
On Linux platform, shellcode is executed by modifying /proc/self/mem. On Windows platform, shellcode is implanted into the process with PID -1 through Java, so as to construct JPLISAgent object and obtain all capabilities of calling Java Agent.
In the BeichenDream's Kcon2021Code project, similar code with this technology idea is also shared.
In the implementation of memory shellcode, a Javassist dependent jar is injected into the target environment without landing, and the target critical class is dynamically modified to inject malicious logic, which realizes the dynamic modification of Agent shellcode. For example, the following figure shows the logic of hooking doFilter method of ApplicationFilterChain, injecting Behinder memory shellcode, and dumping class from the server.
https://preview.redd.it/7l4172y1auqa1.png?width=3468&format=png&auto=webp&s=3677d0f849286b6622da39e2ee7267d904d9a2a8

04 Usage of Memory Shellcode

The problem of exploiting vulnerabilities to directly inject memory shellcodes and the generation and utilization methods of memory shellcodes have been resolved. The next problem to be addressed is the issues encountered during the use of memory shellcodes.
As mentioned in previous articles, the main purpose of the Memory shellcode technology is to combat the problem of security protection devices detecting and alarming against landed files. Therefore, since its inception, Memory shellcode technology has faced and shouldered the responsibility and mission of confronting various protection capabilities.
4.1 Bypassing Security Protections
The first challenge is **bypassing traffic-side devices**. This is actually the traffic characteristics of the communication protocol between the WebShell management side and the memory shell. Since AES encryption and decryption are commonly used, with a small number of cases using DES encryption and decryption, and there are regular behaviors, such as sending several packets when connecting to the WebShell, there are some means to detect webshell connections based on these two factors. Therefore, whether it is the Behinder or Godzilla, if they have not been customized, their basic traffic characteristics will be detected.
However, basically everyone has the habit of customization, so the traffic layer characteristics are still not easy to be uniformly protected, and the latest Behinder client already supports custom communication protocol encryption and decryption programs. This allows attackers to disguise Behinder traffic as similar to business data traffic, such as Restful API return data, or similar base64 image resource return data.
The second challenge is **bypassing host-level protections**. At the host level, there may be some host-level defenses such as EDR devices, which may monitor Java process calls to system resources. However, most of the time, it is almost impossible for this level of defense to determine whether Java-level operations are sensitive operations.
Finally, there is **bypassing Java-level protections**. At the Java level, there may be some RASP products or custom security rules defenses. These defenses intercept suspicious behaviors based on stack or behavior, and hook at the position where some sensitive functions are executed.
At this point, we can bypass these defenses through reflection. Whether it is to call deeper code or even native methods through reflection, or to reflectively obtain objects that encapsulate specific methods in the system for execution, the purpose is to disrupt the stack or behavior call chain, making Java-level defense unable to determine whether you are performing malicious operations or system behaviors, thus bypassing the detection logic.
For example, bypassing command execution defense through reflection to call native methods:
https://preview.redd.it/s2akxi85auqa1.jpg?width=2320&format=pjpg&auto=webp&s=1a84dae44723fba626af08b2c509a783df28be41
Or use messy reflection to make the call chain difficult to trace:
https://preview.redd.it/sdx9g37gauqa1.png?width=2588&format=png&auto=webp&s=b92d429a2196af1b0647347b431bfd6899c77da3
Creating malicious classes using APIs like unsafe can also bypass certain security defenses:
https://preview.redd.it/5wfyjklhauqa1.png?width=2486&format=png&auto=webp&s=d07b6289921110f5511897df95d3c4888f07c786
4.2 Anti-detection
As mentioned in previous memory shellcode articles, many tools have provided detection methods to scan specific locations to check for the presence of memory shellcode. At this time, the check will include some dimension judgments. Similarly, we need to perform certain processing on these dimensions to prevent detection, for example:

1. Detection of malicious class names and package names: For some defense measures, loading of known malicious package names and class names will be prohibited. Therefore, we use dynamic splicing and generation of malicious class package names to confuse the defense system or administrator.

https://preview.redd.it/3v02n0xnauqa1.png?width=1780&format=png&auto=webp&s=0c8d71a8e5639575edffeea9ef89730417079dfe

1. Detection of whether files are written to disk from ClassLoader: The detection logic can be bypassed by using a custom ClassLoader to carry false information or loading malicious classes using the system class loader with an empty class loader for the malicious class.

https://preview.redd.it/km3dlaiuauqa1.png?width=3402&format=png&auto=webp&s=8ee567dd3916cc7ffa05af54465fd7e9ff71d49d

1. Detection of critical positions in the system: Some detection tools can obtain information about critical positions and assist in manual inspection. For example, some tools obtain all Filter-type memory shellcode in the system and display them. At this time, it is possible to evade detection by exploring unconventional memory shellcode. As mentioned in the PPT I shared earlier about JavaWeb memory shellcode, all components that use the chain of responsibility design pattern in the web request processing process can be used as directions for exploring and utilizing memory shellcode. Therefore, it is not difficult to explore a new type of memory shellcode in various web middleware.

​
https://preview.redd.it/i6k57idwauqa1.png?width=2414&format=png&auto=webp&s=26573c1053879ba3d680b3dc44140da1f410ac09

1. Many tools offer the ability to dump the class, allowing for troubleshooting by dumping the class bytecode in memory. Therefore, it is possible to modify the cache of relevant information in the InstanceKlass data structure of the Java class in the JVM, such as _cached_class_file, to deceive and hide by making the dumped class not contain dangerous code.
   
2. Some RASPs also use redefineClasses to set the critical method content of malicious classes and functions to empty, in order to clear the memory shellcode in the running system. At this point, it is possible to make it fail by modifying the function modifiers, adding member variables, methods, etc. of the malicious class, as redefineClasses does not allow changes in class structure and signatures.
   
3. Currently, most of the methods for detecting and defending against memory shellcode are implemented through Java Agent technology. Therefore, preventing new Agent injections is also a key strategy for preventing detection. As mentioned in the first article, blocking the communication between JVM processes by deleting the java pid file and preventing the loading of subsequent ClassFileTransformers can prevent the loading of other Java Agents and prevent detection.
   

​
4.3 Disappear Without a Trace
First of all, since memory shellcode have reached the point of not leaving files behind, is there anything else that can be done to hide themselves again? The answer is yes.
That is, clearing the access logs of middleware. When making access requests, middleware records logs, which are usually used as the basis for subsequent reviews and emergency responses. If access logs can be cleared during memory shellcode access, wouldn't that be anonymous browsing?
With the idea in place, the execution is simple, which is to find the component responsible for logging in the middleware and clear it. Taking Tomcat as an example.
​
https://preview.redd.it/hfr744p2buqa1.png?width=3482&format=png&auto=webp&s=c446370ad9a73d31dd86cc8c23ffeddfbf659606
4.4 Persistence
The final issue is the issue of persistence, which needs to consider whether the injection of memory shellcode can be restored after service restart or even operating system restart:

• For Java, Java shutdown hook can be used for landing and other operations of memory shellcode. If the target environment is Tomcat, JSP files can be written in the resource directory of Jar package, etc.;
• If the target environment may be killed by -9, a "daemon process" can be started to monitor the Java process on the server;
• For operating system restart, critical malicious operations can be registered as timed tasks in advance to achieve persistence.

Since these actions are an extension of memory shellcode technology and may involve tampering and landing of Jar packages and resource files in order to achieve persistence, which is somewhat contrary to the original intention of using memory shellcode, this part will not be discussed further, and we look forward to more elegant ideas.

05 Summary

The above section briefly lists some technical issues and solutions encountered in practical use of memory shellcode technology. After researching and resolving the above techniques, there should be no problem in using memory shellcode quickly in practice.
Although we are discussing JavaWeb memory shellcode technology, it can be seen that the thinking and technology of the countermeasures have already extended beyond the Java layer to the native layer and memory level. This is still a drop in the bucket in practical use. In actual use, due to differences in operating systems, middleware versions, JDK distributions and versions, security restrictions, security protection and other complex situations, there will be various difficulties. Therefore, more research and debugging, and accumulation of ideas can enable efficient and fast use of memory shellcode in practical use.
In the face of memory shellcode technology, it is superficially a technical confrontation, but in fact it is a confrontation between people and people, thinking and thinking. I throw out some ideas here, hoping to inspire more ingenious ideas, and welcome everyone to discuss.
​
[All articles in the memory shellcode series](https://github.com/gobysec/Memory-Shell)