1. sudo pacman -S openconnect networkmanager-openconnect
2. Gateway: anyc.vpn.gatech.edu
3. IPv4->Routes->Only use for resources on this network
4. sudo setcap cap_net_admin+ep /usr/bin/openconnect
5. sudo nano /usr/share/dbus-1/system.d/nm-openconnect-service.conf, edit the last two denys into allow.
6. Profit!

This is my 2^nd time participating in a CTF contest. This time I paired up with awesome people from TUNA and THU CSTA in the Tea Deliverers team. These guys are like superheroes when it comes to problem-solving!

Finally I was only able to fully solve one problem: poly. I'll show the problem and my solution below.

Problem

nc poly.ctfcompetition.com 1337

It will present this banner:

Give me a bios image that I can load into qemu that will send
"Pull_that_lever1"to 169.254.169.254:80 over tcp. The server can
be delayed a bit, so your payload should retry connecting. And by
the way, it has to work both in x86_64 and on a qemu arm virt-2.8
machine, all in under 300 seconds

To make your life easier, here are the commands the server will run:

/usr/bin/qemu-system-arm -nographic -machine virt-2.8 -net nic -net "user,restrict=on,net=169.254.0.0/16,host=169.254.169.253,guestfwd=tcp:169.254.169.254:80-cmd:netcat 127.0.0.1 $random_port" -bios $tmpfile

/usr/bin/qemu-system-x86_64 -nographic -net nic -net "user,restrict=on,net=169.254.0.0/16,host=169.254.169.253,guestfwd=tcp:169.254.169.254:80-cmd:netcat 127.0.0.1 $random_port" -bios $tmpfile

Boot image (base64 encoded, limit 3MiB encoded):

It is obvious that it wants you to construct a polyglot that can run
both on ARM and x86. At first thought one would think of using some
"dual-use" instructions to construct a "loader" and jump to payloads
for different platforms. This strategy works for shellcodes beautifully, however, this will not work because we are constructing a BIOS image.

BIOS Loading Process on x86 and ARM

Let there be some background information on the BIOS loading process of an x86 machine.
We all know that the BIOS is actually stored on some flash memory chip on an actual motherboard. On x86 this memory is mapped to the 0xF0000 to 0x100000 range. The key point is that, the system will jump to 0x100000 after initialization. And this address is actually the end of the BIOS ROM image.

0x00000 .. 0xA0000      DOS Memory Area       RAM
0xA0000 .. 0xC0000      Video Memory          Device Memory
0xC0000 .. 0xE0000      ISA Extension ROM     ROM
0xE0000 .. 0xF0000      BIOS Extension ROM    ROM
0xF0000 .. 0x100000     BIOS Area             ROM
(Quoted from QEMU Wiki)

However, on ARM machines, this is not the case. The address mapping is largely vender-dependent while for our case (The QEMU virt 2.7 machine), Execution starts from 0x00000000 and this is mapped to the begin of the ROM image. Voila! We just need to concat the two files and our polyglot is done!

Solution

Now the problem becomes "how to create the payloads for ARM and x86?". Well, there is a project called U-Boot (Das U-Boot) that is known by every embedded developer, which is a bootloader for Linux (and others) for almost all platforms. We just compile U-Boot on both platforms and concat them! Also, your final payload must be padded to multiples of 65536 bytes.

And now we have the following snippet:

; multi-arch u-boot

; compile with nasm
bits 32
_start:
    incbin "../u-boot/u-boot.bin"
    times 23812 db 0x90 ; This is calculated padding
    incbin "../u-boot-x86/u-boot.rom"

Now the payload will already be running on both platforms. Meanwhile, another problem arises that the two machines have different Ethernet hardware: e1000 for x86 and virtio-net for ARM. Fortunately, U-Boot has built-in support for both cards. We just need to compile it and add some code for actually loading it with the U-Boot Driver Model (DM).

int do_demo_net(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
{
	struct udevice *dev;
	int i, ret;

	puts("VIRTIO uclass entries:\n");
	int devnum = 0;
	struct udevice *net_dev;
	ret = uclass_get_device(UCLASS_VIRTIO, devnum, &net_dev);
	printf("getting devices %d\n", ret);
	for (i = 0, ret = uclass_first_device(UCLASS_VIRTIO, &dev);
	     dev;
	     ret = uclass_next_device(&dev)) {
		printf("entry %d - instance %08x, ops %08x, platdata %08x\n",
		       i++, (uint)map_to_sysmem(dev),
		       (uint)map_to_sysmem(dev->driver->ops),
		       (uint)map_to_sysmem(dev_get_platdata(dev)));
	}

	if (ret == 0) {
		ret = uclass_get_device(UCLASS_ETH, devnum, &net_dev);

		printf("getting devices %d\n", ret);
		for (i = 0, ret = uclass_first_device(UCLASS_ETH, &dev);
			dev;
			ret = uclass_next_device(&dev)) {
			printf("entry %d - instance %08x, ops %08x, platdata %08x\n",
				i++, (uint)map_to_sysmem(dev),
				(uint)map_to_sysmem(dev->driver->ops),
				(uint)map_to_sysmem(dev_get_platdata(dev)));
		}
	}
	return cmd_process_error(cmdtp, ret);
}

Seems easy, right? It comes with a big surprise-the U-Boot network stack does not support TCP! I patched it against a TCP patch that I found from the U-Boot mailing list, and hacked it up to send the challenge and receive the reply(Turns out to be unnecessary).

Now we just pwn the server:

from pwn import *

#context.log_level = 'debug'
context.proxy = (socks.SOCKS5, 'localhost', 1080) 
p = remote("poly.ctfcompetition.com", 1337)

p.recvuntil("(base64 encoded, limit 3MiB encoded):")
p.send(file("./noptest.b64", "rb").read())
p.interactive()
exit()

And here be the flag!

P.S. & Acknowledgment

Really exciting competition and a nice learning experience for a 菜鸡 (newbie) like me.

The patched U-Boot with TCP can be found at https://github.com/ProfFan/u-boot.
Check the commit logs to learn how to use it.

The author would like to thank:

• The U-Boot authors :)
• The patch author Duncan Hare for the TCP Patch
• @riatre for his expertise (waaaaaaay above me XD)
• @iromise for inviting me to the team
• TUNA and all people from THU CSTA

TS;JR

Basically you can just follow the steps in https://www.youtube.com/watch?v=OvcGn_ScG5w

However, at the step when you need to generate the keys with ./rigup license [keyfile] 0x1C0FF, do not just run the command. Run cat [keyfile], and compare with the device serial you obtained by SCPI IDN command. You will discover that the serial number is wrong.

rigup scan - Version 0.4.1

        Hacked up for MSO1000Z(-S) rmd79, 0ff eevblog.com

RC5KEY1:        9BBBBBBBBBBBBBBBBBBBBBBBBBFAD3E6
RC5KEY2:        8BBBBBBBBBBBBBBBBBBBBBBBBB0A699C
XXTEAKEY:       8BBBBBBBBBBBBBBBBBBBBBBBBB24850B
PUBKEY:         00XXXXXXXXXXXXXX
PRIVKEY:        00XXXXXXXXXXXXXX
SERIAL:         DS1ZA203000000 [ This is WRONG ]

Replace the serial number with the one you got using SCPI *IDN?.

Now you can generate the license with ./rigup [keyfile] mso1100z.txt 0x1C0FF.

And install with SCPI :SYSTem:OPTion:INSTall Y4HHHHW66SW9HHHHMU75BBBBBBBB.

Voilà! Now you have a scope with all options unlocked.

What is a Smartcard

A smartcard is a portable device with a MCU (usually security-enhanced MCU) that can run programs stored in its secure memory, for example cryptographic algorithms. In other words, a smartcard is a "secure world" where you can store sensitive information, which systems outside of the card can use the information for authentication and encryption without revealing the secret stored.

What will you need

• A Smartcard
  • JavaCard-based (Mine is JC30M48CR)
  • A guide can be found here
• A Smartcard Reader
  • Any standard smartcard reader
  • Rule of thumb is to buy those can read SIM cards
  • Mine is Rocketek RT-SCR2 (Alcor Micro AU9560)
• Latest GnuPG (GPG)

Great thanks to Fan Dang for kindly providing me with a card, you can visit his blog here.

Preparing the Card

First you need to install the following packages:

sudo pacman -S ant maven pcsclite pcsc-tools

Also make sure that you have OpenJDK 10 installed and selected with archlinux-java set java-10-openjdk

And install GlobalPlatformPro by Martin Paljak:

git clone https://github.com/martinpaljak/GlobalPlatformPro.git
cd GlobalPlatformPro
mvn package && ant

Now you can peek into your card with:

$ java -jar target/gp.jar -i

GlobalPlatformPro 19.01.22-3-g05a90c2
Running on Linux 5.0.9-arch1-1-ARCH amd64, Java 10.0.2 by Oracle Corporation
Reader: Alcor Micro AU9560 01 00
ATR: 3B90958011FE6A
More information about your card:
    http://smartcard-atr.appspot.com/parse?ATR=3B90958011FE6A

Card Data:
Card Capabilities:
Version: 255 (0xFF) ID:   1 (0x01) type: DES3 length:  16
Version: 255 (0xFF) ID:   2 (0x02) type: DES3 length:  16
Version: 255 (0xFF) ID:   3 (0x03) type: DES3 length:  16
Key version suggests factory keys

If you have multiple card readers, just add -r "Alcor Micro AU9560 01 00" to the command line (replace with your reader name).

> java -jar target/gp.jar -l

Warning: no keys given, using default test key 404142434445464748494A4B4C4D4E4F
ISD: A000000003000000 (OP_READY)
     Privs:   SecurityDomain, CardLock, CardTerminate, CardReset, CVMManagement

Now you can see that we have an empty card with default keys. Notice that this key is just like the root password of your system, so PLEASE CHANGE IT TO A SECURE ONE before using it as a secure device!

Now let's change the card key. Notice that this key is necessary for doing any administrative task on the card, so please keep it stored somewhere SAFE and DO NOT FORGET it.

First generate a true random secret with OpenSSL:

openssl rand --hex 16 | awk '{print toupper($0)}' > .card_secret

Now we can lock the card with this key:

cat .card_secret | xargs java -jar target/gp.jar -r "Alcor Micro AU9560 01 00" -lock

Warning: no keys given, using default test key 404142434445464748494A4B4C4D4E4F
Card locked with: D61B707C088B2F8829C7218FC839A664
Write this down, DO NOT FORGET/LOSE IT!

FROM THE POINT YOU LOCKED YOUR CARD, ALL COMMANDS MUST TAKE THE ARGUMENT -key {YOUR_SECRET} OR YOU MAY BRICK YOUR CARD!

However it is a great hassle if we need to add the key to our command each time, so we can first unlock the card, if we can remember to LOCK BEFORE USING THE CARD AS SECURE!

Now we unlock the card with:

cat .card_secret | xargs java -jar target/gp.jar -r "Alcor Micro AU9560 01 00" -unlock -key

Default type=DES3 bytes=404142434445464748494A4B4C4D4E4F kcv=8BAF47 set as master key for A000000003000000

Again, REMEMBER TO LOCK THE CARD BEFORE PRODUCTION!

Building SmartPGP

SmartPGP is an open-source implementation of the OpenPGP card specification, which works with GnuPG for secure signing and encryption.

To build SmartPGP, first clone the SmartPGP repository:

git clone https://github.com/ANSSI-FR/SmartPGP.git
git checkout javacard-3.0.1 # Only this version fits in 80K of memory
# If your card has larger Flash, just use the master branch
# If master fails to build, checkout this branch and retry

And add the JavaCard SDK:

git submodule add https://github.com/martinpaljak/oracle_javacard_sdks sdks

Now you need to change the config parameters for SmartPGP.

First specify the SDK path with jckit= in build.xml:

    <javacard jckit="./sdks/jc304_kit">
      <cap output="SmartPGPApplet.cap" sources="src" aid="d27600012401" version="1.0">
        <applet class="fr.anssi.smartpgp.SmartPGPApplet" aid="d276000124010303AFAF000000000000"/>
      </cap>
    </javacard>

The original SmartPGP requires too much on-card RAM, which will lead to errors when loading the applet. To reduce the RAM usage, change src/fr/anssi/smartpgp/Constants.java:

    protected static final short INTERNAL_BUFFER_MAX_LENGTH =
        (short)0x300;

Now we can start building the applet by:

ant

which will create the applet CAP file SmartPGPApplet.cap.

Installation of the Applet

Now it is time to install the applet to your card:

java -jar target/gp.jar -r "Alcor Micro AU9560 01 00" -install ../SmartPGP/SmartPGPApplet.cap

Warning: no keys given, using default test key 404142434445464748494A4B4C4D4E4F
CAP loaded

If it shows

Warning: no keys given, using default test key 404142434445464748494A4B4C4D4E4F
CAP loaded
INSTALL [for install and make selectable] failed: 0x6985 (Conditions of use not satisfied)

Your card do not have enough RAM and you need to decrease the INTERNAL_BUFFER_MAX_LENGTH parameter.

If it shows

Warning: no keys given, using default test key 404142434445464748494A4B4C4D4E4F
Loading failed. Are you sure the CAP file (JC version, packages, sizes) is compatible with your card?
LOAD failed: 0x6A80 (Wrong data/incorrect values in data)

Your card do not support the JavaCard SDK version. Please try a lower version of the SDK, e.g. jc304_sdk if you are using jc305u1_sdk or higher.

Now you should LOCK YOUR CARD by:

cat .card_secret | xargs java -jar target/gp.jar -r "Alcor Micro AU9560 01 00" -lock

Setting up the card

Just follow the Debian wiki at https://wiki.debian.org/Smartcards/OpenPGP.

If you want to use the card for system login, try Poldi.

If you have questions or problems, please ask in the comments area below.

Wish you a nice journey playing with smartcards!

Reference

A list of JavaCard goodies can be found at https://github.com/EnigmaBridge/javacard-curated-list

Usage is simple: First clone the repo. Install Rust using rustup and compile. Install diesel.rs and setup PostgreSQL. Edit the .env file to provide DB information to the system and diesel setup. And now you are good to go!

Then create a user, a site and give the user privilege:

cargo run --bin new_user

cargo run --bin new_site

cargo run --bin new_priv

And run the web app with:

ROCKET_CFUSER=<CF_EMAIL> ROCKET_CFKEY=<CF_KEY> cargo run

To call the API, simply use curl:

curl --verbose  --header "Content-Type: application/json" \
  --request POST \
  --data '{"user":"<USER>","key":"<API_KEY>","zone":"amayume.net","rec":"py.amayume.net","rectype":"A", "value":"119.44.22.22"}' \
  http://ddns.amayume.net:1337/update

Hope it can be useful for you :)

Difference in Normal C++ and UE4 C++

CDO v.s. Constructor

Basically the constructor of a C++ UObject is called in the following cases:

• Engine Start: construction of CDO, in FEngineLoop::PreInit
• BP Loading: also CDO, in UClass::Serialize (need confirm)
• Object Loading(Spawn): normal constructor

To determine the current mode, use:

if (HasAnyFlags(RF_ClassDefaultObject)) {
    // Do CDO
} else {
    // Do normal init
}

So CDO is just for initializing of the UPROPERTY fields? Need to consider in design.

This is possibly the reason why the current version has so many bugs :)

BTW, for normal objects, just newing it is sufficient. So CDO only applies to UObjects.

GC

Big headache for me. UE4 by default garbage-collect everything that is too far away. For example, if you have a physics-enabled object that does not have collision with the ground, it will fall down infinitely, and then GC'ed as out-of-range. You may think this is not important, right? However think that when only the root component is set like that and it falls down too much-all other components behaves normally and all of a sudden your actor disappears entirely. Such situations are like nightmare for debugging-they are actually "normal" behaviors of the engine, thus leave no logs at all.

Another problem for GC is that it relies on references. And references rely on Unreal's Object System. If an object is not declared as UObject, it will not be considered in GC, which will result in memory leaks. Furthermore, if the non-UObject object refer to a GC'ed UObject, there will be a dangling pointer.

It is noteworthy that, Unreal Objects are also C++ objects. One may think, if we only use smart pointers in UObject for unmanaged objects and not refer to UObjects in unmanaged C++ code, the problem can be avoided in some cases.

However, consider the following situation: we have an UObject that needs to do some heavy lifting using a third-party library asynchronously. To do that, we call a function with a callback function as argument. Before the work is done, the UObject gets released by GC. Now the callback function becomes a dangling pointer. This example explains why it is impossible to avoid this problem in modern, async-heavy applications.

Interoperation Patterns

An RPC-like solution is always nice to have. For example, the ongoing effort on integrating Cap'n Proto will enabled cross-thread and cross-process communication both in Unreal and between other components.

Cap'n Proto is nice in that it is message loop and Promise based. In this way exceptions can be handled without using exceptions. It also supports -fno-rtti. However the internals of Cap'n Proto is not well documented due to its relatively small user base. Still needs investigation.

ROS-based solutions can directly communicate with ROS without a bridge. However, they are usually deeply plagued by Boost, RTTI and other anti-patterns. The only standalone ROS library is cROS, which is limited in function and does not have a nice API.

[TODO: figure out the final architecture and start rebuilding Pavilion.]

In the process of implementing that concept, I have grown up as a half-professional Unreal developer-yet the project is still a big mess with tons of spaghetti-like code. This project finally made me realize that I am not omnipotent. I cannot single-handedly make a huge project work like magic in such a short time.

In retrospect, the current code base is non-maintainable to say the least. Functionalities are not clearly separated to modules, and modules have too much inter-dependencies that they have to be pulled together when you just need one, just like the famous-and-infamous Boost. Exceptions seems unavoidable. RTTI can only be eliminated by manual effort, or you will have to enable RTTI for the entire Unreal Engine.

RTTI

Who is to blame for this situation? RTTI is a controversial topic in C++ since its advent. It's sort of like GPL in the sense that it "infects" all the libraries that uses it, whether its shared (dynamically-linked) or static. The only way out is to expose all APIs in C, which then makes object-oriented programming a joke.

However, RTTI is not evil. It is indeed useful in many sense. It is just broken in implementation. However, many programmers actually misuse RTTI for cases where a virtual function would suffice. If that is not the case, LLVM-style RTTI would in 99.9% of the cases, which does not require the C++ RTTI system to be on. Sadly, most C++ programmers simply takes the short path and enable RTTI, without considering the consequences.

Exceptions

Exceptions are said to have "zero-cost" in modern C++. Yet we know that nothing in this world comes truly with no cost. Exceptions adds a lot of footprint to your binary. Misuse of exceptions creates alternate code paths that is implicit, that makes your codebase bug-prone. Throwing in constructors is also a highly controversial issue.

At least 50% of the use of exceptions in C++ can be replaced without using it. For example, a throw in constructor when resource acquisition can be replaced for a non-throwing constructor and a throw in subsequent calls. Or even returning a bool in a init() method.

class A {
public:
    A() {
        bool connected = false;
        // connect to a server
        if (!connected) {
            throw NotConnectedException();
        }
    }
}

class B {
    bool connected = false;
public:
    B() {
        // connect to a server
    }
    
    void send() {
        if(!connected) {
            throw NotConnectedException();
        }
    }
}

class C {
    bool connected = false;
public:
    C() {
        
    }
    
    bool init() {
        // connect to a server
        return connected;
    }
}

As you can see in the above example, class C entirely avoided the use of exceptions, while abide by the RAII (Resource-Acquisition-Is-Initialization) principle. Again, the problem is that most programmers write code that works, not code that works for everyone.

LLVM is great. Yes. But LLVM is full of caveats and especially when it comes to documentation. Most of the docs you can find on the Internet is for LLVM 3.x (3.5, 3.9) and 4.x. LLVM is now 6.x and 7.x is on the way to release. Hopefully this post can help you solve some of your problems in learning LLVM and/or migration of LLVM versions.

LLVM IR and CUDA Related

LLVM IR is an interesting language: it is used by lots of projects as an intermediate language, however it is in no way stable. The IR specification changes every release of LLVM and have no guaranteed compatibility.

This problem becomes even worse with the proprietary NVVM compiler, the de facto compiler for CUDA. NVVM is still using the LLVM 3.4 version as its base (as of CUDA 9.2), as you can see here:

//
// Generated by NVIDIA NVVM Compiler
//
// Compiler Build ID: CL-23757830
// Cuda compilation tools, release 9.2, V9.2.64
// Based on LLVM 3.4svn
//

Thus it will not be able to read in the IR generated by LLVM with any version larger than 4.0. What made it worse is that there is actually no tool to convert between IR versions. This bites me when I was porting the CUDA functionality of Stanford's Terra language to LLVM 6.0. Every single function call succeeded without a warning, however the program does not work at all! And when I checked the output of NVVM, it is not empty. No. It is comment-only! Kind of like I was completely fooled by the non-empty string size.

To solve this incompatibility, we have to resort to the open-source counterpart of NVVM, the NVPTX backend of LLVM. It is not tuned by folks at NVIDIA, hence possibly lower performance. It is relatively easy to use, with one page of documentation that has plenty of information. Just you need to link the libdevice yourself, or you will get errors when doing anything with a math function.

LLVM API Related

The LLVM API, just like the IR, is also not stable between releases.

Linker API

Almost all the information you can find on the Internet is like this:

llvm::Linker linker("clang_test", "clang_test", lc, llvm::Linker::Verbose);

std::string error;
if( linker.LinkInModule( new_module, &error ) || !error.empty() ) {
    printf( "link error\n" );
    return -3;
}

llvm::Module* composite_module = linker.getModule();
if( composite_module == NULL ) {
    printf( "link error\n" );
    return -3;
}

Which is not correct for LLVM >3.4. In fact, LLVM now standardized all APIs to be Module-centric, i.e., everything is modular and these strings are gone. This means a little bit more boilerplate code but a lot more grasp for the user on understanding what is going on under the hood.

See it yourself:

auto libdevice = "/usr/local/lib/libdevice.bc"; // The library you want to link in
llvm::SmallString<2048> ErrMsg;
auto MB = llvm::MemoryBuffer::getFile(libdevice); // Get the contents of the lib file
auto E_LDEVICE = llvm::parseBitcodeFile(MB->get()->getMemBufferRef(), M->getContext()); // Detroit: Become Module

if (auto Err = E_LDEVICE.takeError()) { // New error handling
    llvm::logAllUnhandledErrors(std::move(Err), llvm::errs(), "[CUDA Error] ");
    return;
}

auto &LDEVICE = *E_LDEVICE;

auto TargetMachine = Target->createTargetMachine("nvptx64-nvidia-cuda", cpuopt, Features, opt, RM);

LDEVICE->setTargetTriple("nvptx64-nvidia-cuda");
LDEVICE->setDataLayout(TargetMachine->createDataLayout());

llvm::Linker Linker(*M); // M is the reference of the module you want to link libdevice in
Linker.linkInModule(std::move(LDEVICE));

As you can see above, now you need to first read in and parse the module to get a unique_ptr to the llvm::Module you want to link in, and create a llvm::Linker with the target module M, then link the library in by calling linker.linkInModule. Also note that how LLVM is now using smart pointers to handle memory: previously there are APIs like  linker.releaseModule, and now they are completely history.

To be continued :)

There's been something lurking in my mind for years–something that can be described as a dream, or more precisely, as an objective. It is resurrected every time I recalled the biology seminars I took. I can remember all the fierce firefight with professors and fellow students over my slides and presentations. Every seminar session is a battlefield with people desperately searching for logical errors, data inconsistencies, misleading figures, and even grammar mistakes, to falsify the presenter's work.

After I entered the field of computer science and engineering, things become much different. In nearly every single seminar I attended, I can only see peace and friendly discussion. There's been no one asking tough questions. In a biologist's eyes, tons of deep learning papers lacks enough evidence to prove their conclusion. Yet they got published into top conferences and journals. It seems that people nowadays only cares about performance.

Learning deep learning concretely is merely a distant dream.

I then made a stupid mistake–I deleted the drivers entirely, a.k.a. all relevant files in C:\WINDOWS\INF, C:\WINDOWS\System32\drivers and most importantly, C:\WINDOWS\System32\drivers\DriverStore. After that I run sfc again and it still does not work!

After cursing Microsoft for making such a silly system repair utility, I return to try to manually diagnose the issue. I first manually copied back all files to INF and drivers, but the driver installations still did not go through. I then try putting back the DriverStore folders–then I found out that I completely forgot their proper names!

Normal Windows Event Viewer only shows a 0x2 error code, so I googled tons of pages to find the location of a proper setup log. It was two hours later that I found the location: C:\Windows\Inf\setupapi.dev.log

After that things went smoothly. You just need to search for Error 2 in the log, and the proper path revealed itself.

!!!  flq:                     Source Media: SPFQOPERATION_ABORT (C:\Windows\System32\DriverStore\FileRepository\wdmaudio.inf_amd64_179dade7dec66387\drmk.sys).
!!!  flq:                     Error 2: The system cannot find the file specified.
!!!  flq:                     FileQueueCommit aborting!
!!!  flq:                     Error 2: The system cannot find the file specified.

Now you just need to copy the directory in WinSxS to the DriverStore and rename. After that the audio drivers are finally back working.

Parsing

Parsing is mostly documented in the previous post. However the resulting data structure is Pest's own Iter-based format. We need to convert it into an intermediate representation with the Serialize trait required by handlebars-rust. This is done via a custom ASTDef type with the required traits. Though I think there may be better methods to do that, this is the best I can come up with.

#[derive(Serialize, Deserialize, Debug)]
pub struct ConstDef {
    typ: String,
    name: String,
    val: String,
    comment: String,
}

#[derive(Serialize, Deserialize, Debug)]
pub struct FieldDef {
    typ: String,
    name: String,
    comment: String,
}

#[derive(Serialize, Deserialize, Debug)]
pub struct ASTDef {
    consts: Vec<ConstDef>,
    fields: Vec<FieldDef>,
}

fn pest_to_ast(pes: &pest::iterators::Pair<Rule>) -> Option<ASTDef> {
    let mut ast = ASTDef {
        consts: Vec::new(),
        fields: Vec::new(),
    };

    // Because ident_list is silent, the iterator will contain idents
    for pair in pes.clone().into_inner() {
        let span = pair.clone().into_span();
        match pair.as_rule() {
            Rule::constant => {
                let mut typ: Option<String> = None;
                let mut name: Option<String> = None;
                let mut val: Option<String> = None;
                let mut comment: Option<String> = None;

                for inner_pair in pair.clone().into_inner() {
                    let inner_span = inner_pair.clone().into_span();
                    match inner_pair.as_rule() {
                        Rule::itype => {
                            typ = Some(inner_pair.as_str().to_string());
                        }
                        Rule::identifier => {
                            name = Some(inner_pair.as_str().to_string());
                        }
                        Rule::value => {
                            val = Some(inner_pair.as_str().to_string());
                        }
                        Rule::comment => {
                            comment = Some(inner_pair.as_str().to_string());
                        }
                        _ => panic!("ERR {:?}:   {}", inner_pair.as_rule(), inner_span.as_str()),
                    };
                }

                ast.consts.push(ConstDef {
                    typ: typ.expect("No type detected"),
                    name: name.expect("No name detected"),
                    val: val.expect("No val detected"),
                    comment: comment.unwrap_or("".to_string()),
                })
            }
            Rule::variable => {
                ...
            }
            Rule::comment => {
                // println!("{}", pair.as_str());
            }
            _ => panic!("ERR {:?}:   {}", pair.as_rule(), span.as_str()),
        }
    }

    return Some(ast);
}

As you can see, every node needs to be filled manually. If you have a better method, please comment below :)

Code Generation

The Cap'n Proto code is then generated by handlebars-rust. Because we implemented the Serialize trait in our AST, we can now simply use &serde_json::to_value(&ast) to feed JSON data to handlebars-rust.

fn compile_file(ast: ASTDef) {
    use handlebars::Handlebars;

    let reg = Handlebars::new();

    let template =
r###"struct BatteryState {
{{#each consts as |c| ~}}
  const {{c.name}} @0 : {{c.typ}} = {{ c.val }};{{ c.comment }}
{{/each ~}}

{{#each fields as |f| ~}}
  {{f.name}} @0 : {{f.typ}};{{ f.comment }}
{{/each ~}}
}"###;
    // render without register
    println!("{}", reg.render_template( template, &serde_json::to_value(&ast).unwrap()).unwrap());
}

Conclusion

This project is fairly simple and really nice for a Rust newcomer like myself. Next step would probably be writing the roscpp compatibility layer in Rust and export with C++. Anyway, stay tuned :)

Code

The code is hosted at https://github.com/CoreRC/rcgenmsg. For now it's only a parser adapted from the official pest documentation. The parser definition is listed below:

// ROS Message Definition Parser
// ==========================

file = {
    (result)*
}
result = _{
    (" " | "\t")* ~ (comment | definition)? ~ sp ~ comment? ~ linebreak+
}

sp = _{
    (" " | "\t")*
}

linebreak = _{
    "\n" | "\r"
}

types = {
    ("bool" | "uint8" | "float32" | "string" | "Header" | !("\n" | "\r" | "\t")+)
}

array = {
    types ~ ("[]")
}

itype = {
    array | types
}

identifier = {
    ('a'..'z' | 'A'..'Z' | "_")+
}

value = {
    ('a'..'z' | 'A'..'Z' | '0'..'9')+
}

constant = {
    itype ~ sp ~ identifier ~ sp ~ "=" ~ sp ~ value
}

variable = {
    itype ~ sp ~ identifier ~ sp ~ !("=")
}

definition = {
    (variable | constant)
}

nonl = _{ !linebreak ~ any }
comment = { "#" ~ nonl* }

Test

We tested it with the following ROS msg definition file with constants, variables, arrays and inline comments:

# Constants are chosen to match the enums in the linux kernel
# defined in include/linux/power_supply.h as of version 3.7
# The one difference is for style reasons the constants are
# all uppercase not mixed case.

# Power supply status constants
uint8 POWER_SUPPLY_STATUS_UNKNOWN = 0
uint8 POWER_SUPPLY_STATUS_CHARGING = 1
uint8 POWER_SUPPLY_STATUS_DISCHARGING = 2
uint8 POWER_SUPPLY_STATUS_NOT_CHARGING = 3
uint8 POWER_SUPPLY_STATUS_FULL = 4

# Power supply health constants
uint8 POWER_SUPPLY_HEALTH_UNKNOWN = 0
uint8 POWER_SUPPLY_HEALTH_GOOD = 1
uint8 POWER_SUPPLY_HEALTH_OVERHEAT = 2 # test
uint8 POWER_SUPPLY_HEALTH_DEAD = 3
uint8 POWER_SUPPLY_HEALTH_OVERVOLTAGE = 4
uint8 POWER_SUPPLY_HEALTH_UNSPEC_FAILURE = 5
uint8 POWER_SUPPLY_HEALTH_COLD = 6
uint8 POWER_SUPPLY_HEALTH_WATCHDOG_TIMER_EXPIRE = 7
uint8 POWER_SUPPLY_HEALTH_SAFETY_TIMER_EXPIRE = 8

# Power supply technology (chemistry) constants
uint8 POWER_SUPPLY_TECHNOLOGY_UNKNOWN = 0
uint8 POWER_SUPPLY_TECHNOLOGY_NIMH = 1
uint8 POWER_SUPPLY_TECHNOLOGY_LION = 2
uint8 POWER_SUPPLY_TECHNOLOGY_LIPO = 3
uint8 POWER_SUPPLY_TECHNOLOGY_LIFE = 4
uint8 POWER_SUPPLY_TECHNOLOGY_NICD = 5
uint8 POWER_SUPPLY_TECHNOLOGY_LIMN = 6

Header  header
float32 voltage          # Voltage in Volts (Mandatory)
float32 current          # Negative when discharging (A)  (If unmeasured NaN)
float32 charge           # Current charge in Ah  (If unmeasured NaN)
float32 capacity         # Capacity in Ah (last full capacity)  (If unmeasured NaN)
float32 design_capacity  # Capacity in Ah (design capacity)  (If unmeasured NaN)
float32 percentage       # Charge percentage on 0 to 1 range  (If unmeasured NaN)
uint8   power_supply_status     # The charging status as reported. Values defined above
uint8   power_supply_health     # The battery health metric. Values defined above
uint8   power_supply_technology # The battery chemistry. Values defined above
bool    present          # True if the battery is present

float32[] cell_voltage   # An array of individual cell voltages for each cell in the pack
                         # If individual voltages unknown but number of cells known set each to NaN
string location          # The location into which the battery is inserted. (slot number or plug)
string serial_number     # The best approximation of the battery serial number

And the result is here:

Processing: hello.txt
LINE:    # Constants are chosen to match the enums in the linux kernel
LINE:    # defined in include/linux/power_supply.h as of version 3.7
LINE:    # The one difference is for style reasons the constants are
LINE:    # all uppercase not mixed case.
LINE:    # Power supply status constants
LINE:    uint8 POWER_SUPPLY_STATUS_UNKNOWN = 0
PART itype:   uint8
PART identifier:   POWER_SUPPLY_STATUS_UNKNOWN
PART value:   0
LINE:    uint8 POWER_SUPPLY_STATUS_CHARGING = 1
PART itype:   uint8
PART identifier:   POWER_SUPPLY_STATUS_CHARGING
PART value:   1
LINE:    uint8 POWER_SUPPLY_STATUS_DISCHARGING = 2
PART itype:   uint8
PART identifier:   POWER_SUPPLY_STATUS_DISCHARGING
PART value:   2
LINE:    uint8 POWER_SUPPLY_STATUS_NOT_CHARGING = 3
PART itype:   uint8
PART identifier:   POWER_SUPPLY_STATUS_NOT_CHARGING
PART value:   3
LINE:    uint8 POWER_SUPPLY_STATUS_FULL = 4
PART itype:   uint8
PART identifier:   POWER_SUPPLY_STATUS_FULL
PART value:   4
LINE:    # Power supply health constants
LINE:    uint8 POWER_SUPPLY_HEALTH_UNKNOWN = 0
PART itype:   uint8
PART identifier:   POWER_SUPPLY_HEALTH_UNKNOWN
PART value:   0
LINE:    uint8 POWER_SUPPLY_HEALTH_GOOD = 1
PART itype:   uint8
PART identifier:   POWER_SUPPLY_HEALTH_GOOD
PART value:   1
LINE:    uint8 POWER_SUPPLY_HEALTH_OVERHEAT = 2
PART itype:   uint8
PART identifier:   POWER_SUPPLY_HEALTH_OVERHEAT
PART value:   2
LINE:    # test
LINE:    uint8 POWER_SUPPLY_HEALTH_DEAD = 3
PART itype:   uint8
PART identifier:   POWER_SUPPLY_HEALTH_DEAD
PART value:   3
LINE:    uint8 POWER_SUPPLY_HEALTH_OVERVOLTAGE = 4
PART itype:   uint8
PART identifier:   POWER_SUPPLY_HEALTH_OVERVOLTAGE
PART value:   4
LINE:    uint8 POWER_SUPPLY_HEALTH_UNSPEC_FAILURE = 5
PART itype:   uint8
PART identifier:   POWER_SUPPLY_HEALTH_UNSPEC_FAILURE
PART value:   5
LINE:    uint8 POWER_SUPPLY_HEALTH_COLD = 6
PART itype:   uint8
PART identifier:   POWER_SUPPLY_HEALTH_COLD
PART value:   6
LINE:    uint8 POWER_SUPPLY_HEALTH_WATCHDOG_TIMER_EXPIRE = 7
PART itype:   uint8
PART identifier:   POWER_SUPPLY_HEALTH_WATCHDOG_TIMER_EXPIRE
PART value:   7
LINE:    uint8 POWER_SUPPLY_HEALTH_SAFETY_TIMER_EXPIRE = 8
PART itype:   uint8
PART identifier:   POWER_SUPPLY_HEALTH_SAFETY_TIMER_EXPIRE
PART value:   8
LINE:    # Power supply technology (chemistry) constants
LINE:    uint8 POWER_SUPPLY_TECHNOLOGY_UNKNOWN = 0
PART itype:   uint8
PART identifier:   POWER_SUPPLY_TECHNOLOGY_UNKNOWN
PART value:   0
LINE:    uint8 POWER_SUPPLY_TECHNOLOGY_NIMH = 1
PART itype:   uint8
PART identifier:   POWER_SUPPLY_TECHNOLOGY_NIMH
PART value:   1
LINE:    uint8 POWER_SUPPLY_TECHNOLOGY_LION = 2
PART itype:   uint8
PART identifier:   POWER_SUPPLY_TECHNOLOGY_LION
PART value:   2
LINE:    uint8 POWER_SUPPLY_TECHNOLOGY_LIPO = 3
PART itype:   uint8
PART identifier:   POWER_SUPPLY_TECHNOLOGY_LIPO
PART value:   3
LINE:    uint8 POWER_SUPPLY_TECHNOLOGY_LIFE = 4
PART itype:   uint8
PART identifier:   POWER_SUPPLY_TECHNOLOGY_LIFE
PART value:   4
LINE:    uint8 POWER_SUPPLY_TECHNOLOGY_NICD = 5
PART itype:   uint8
PART identifier:   POWER_SUPPLY_TECHNOLOGY_NICD
PART value:   5
LINE:    uint8 POWER_SUPPLY_TECHNOLOGY_LIMN = 6
PART itype:   uint8
PART identifier:   POWER_SUPPLY_TECHNOLOGY_LIMN
PART value:   6
LINE:    Header  header
PART itype:   Header
PART identifier:   header
LINE:    float32 voltage          # Voltage in Volts (Mandatory)
PART itype:   float32
PART identifier:   voltage
PART comment:   # Voltage in Volts (Mandatory)
LINE:    float32 current          # Negative when discharging (A)  (If unmeasured NaN)
PART itype:   float32
PART identifier:   current
PART comment:   # Negative when discharging (A)  (If unmeasured NaN)
LINE:    float32 charge           # Current charge in Ah  (If unmeasured NaN)
PART itype:   float32
PART identifier:   charge
PART comment:   # Current charge in Ah  (If unmeasured NaN)
LINE:    float32 capacity         # Capacity in Ah (last full capacity)  (If unmeasured NaN)
PART itype:   float32
PART identifier:   capacity
PART comment:   # Capacity in Ah (last full capacity)  (If unmeasured NaN)
LINE:    float32 design_capacity  # Capacity in Ah (design capacity)  (If unmeasured NaN)
PART itype:   float32
PART identifier:   design_capacity
PART comment:   # Capacity in Ah (design capacity)  (If unmeasured NaN)
LINE:    float32 percentage       # Charge percentage on 0 to 1 range  (If unmeasured NaN)
PART itype:   float32
PART identifier:   percentage
PART comment:   # Charge percentage on 0 to 1 range  (If unmeasured NaN)
LINE:    uint8   power_supply_status     # The charging status as reported. Values defined above
PART itype:   uint8
PART identifier:   power_supply_status
PART comment:   # The charging status as reported. Values defined above
LINE:    uint8   power_supply_health     # The battery health metric. Values defined above
PART itype:   uint8
PART identifier:   power_supply_health
PART comment:   # The battery health metric. Values defined above
LINE:    uint8   power_supply_technology # The battery chemistry. Values defined above
PART itype:   uint8
PART identifier:   power_supply_technology
PART comment:   # The battery chemistry. Values defined above
LINE:    bool    present          # True if the battery is present
PART itype:   bool
PART identifier:   present
PART comment:   # True if the battery is present
LINE:    float32[] cell_voltage   # An array of individual cell voltages for each cell in the pack
PART itype:   float32[]
PART identifier:   cell_voltage
PART comment:   # An array of individual cell voltages for each cell in the pack
LINE:    # If individual voltages unknown but number of cells known set each to NaN
LINE:    string location          # The location into which the battery is inserted. (slot number or plug)
PART itype:   string
PART identifier:   location
PART comment:   # The location into which the battery is inserted. (slot number or plug)
LINE:    string serial_number     # The best approximation of the battery serial number
PART itype:   string
PART identifier:   serial_number
PART comment:   # The best approximation of the battery serial number

We can see it correctly parsed both normal types and arrays correctly.

Please stay tuned at https://github.com/CoreRC :)

1. 《哥德尔、艾舍尔、巴赫:集异璧之大成》（大名鼎鼎的GEB/MUST READ no matter what）
2. John Stillwell 《数学及其历史》(Math step by step)
3. 《数学桥:对高等数学的一次观赏之旅》(Especially Recommended for Newbies)
4. Felix Klein《高观点下的初等数学(全3册)》(Show you the real magic of math)
5. Kailai Chung 《初等概率论(第4版)(英文版)》(Very good introductory material)
6. Richard Feynman《费恩曼物理学讲义(第1卷)》(You know why)
7. R. Bryant, et al. 《深入理解计算机系统(英文版•第2版)》(MUST READ for CS)
8. R. Graham《具体数学:计算机科学基础(英文版第2版)》(Take this as an appetizer for TAOCP)
9. P. Horowitz, W. Hill《The Art of Electronics》（中文《电子学》 MUST READ for all prospective engineers）
10. Use bookfi.net if you want free PDFs :)

In my make-my-own-ROS project, I want to ensure compatibility with the original ROS ecosystem, so I need to provide the same set of functionality like roscpp. This is can be achieved in a number of steps, of which the first is to convert the message definitions. Initially I want to use Rust for that, but upon discovering an online PEG debugger, I decided to use that first.

Procedure

Really simple. Just write grammar->test->change PEG->test->...

The final version of the PEG definition is posted below:

// ROS Message Definition Parser
// ==========================

file = (result)*
result = sp (comment / definition)* linebreak

_ "whitespace"
  = [ \t\n\r]*

sp "space"
  = $[ \t]*

linebreak = [\n\r]

types "types" = ("bool" / "uint8" / "float32" / "string" / "Header")

array "array" = types ("[]")

type "type" = array / types

identifier "identifier" = $[a-zA-Z_]+

constant "constant" = type sp identifier sp [=] sp [0-9]+

variable "variable" = (type / array) sp identifier sp ![=]
definition "definition" = (variable / constant)

string "string" = sp $[^\n\r]+

comment "comment" = (['#']) string

Result

Thanks to the amazing web-based PEG.js debugger by phrogz, I was able to visualize the result easily in my browser :) As you can see, all tokens are differentiated with different colors. The next step would be translating from grammar tree to Cap'n Proto code!