Skip to content

adding the JsonShapeSerializer #3842

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pulimsr wants to merge 1 commit into
base: main
Choose a base branch
from
Open

adding the JsonShapeSerializer #3842

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion src/aws-cpp-sdk-core/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -543,7 +543,15 @@ check_cxx_source_compiles("
return 0;
}" AWS_HAS_ALIGNED_ALLOC)

add_library(${PROJECT_NAME} ${AWS_NATIVE_SDK_SRC})
add_library(${PROJECT_NAME} ${AWS_NATIVE_SDK_SRC}
include/smithy/client/schema/ShapeSerializer.h
include/smithy/client/schema/CborShapeSerializer.h
include/smithy/client/schema/Schema.h
include/smithy/client/schema/JsonShapeSerializer.h
include/smithy/client/schema/QueryShapeSerializer.h
include/smithy/client/schema/XmlShapeSerializer.h
source/smithy/client/schema/JsonShapeSerializer.cpp
)
add_library(AWS::${PROJECT_NAME} ALIAS ${PROJECT_NAME})

target_compile_definitions(${PROJECT_NAME} PUBLIC "AWS_SDK_VERSION_MAJOR=${AWSSDK_VERSION_MAJOR}")
Expand Down
2 changes: 2 additions & 0 deletions src/aws-cpp-sdk-core/include/smithy/client/schema/Schema.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ enum class ShapeType : uint8_t {
class Schema {
public:
Schema() = default;
Schema(const char* memberName, ShapeType type)
: m_type(type), m_memberName(memberName) {}

ShapeType GetType() const { return m_type; }
const char* GetId() const { return m_id; }
Expand Down
209 changes: 209 additions & 0 deletions src/aws-cpp-sdk-core/source/smithy/client/schema/JsonShapeSerializer.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,209 @@
/**
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
* SPDX-License-Identifier: Apache-2.0.
*/
#include <aws/core/utils/HashingUtils.h>
#include <aws/core/utils/json/JsonSerializer.h>
#include <smithy/client/schema/JsonShapeSerializer.h>

using namespace smithy::schema;
using namespace Aws::Utils;
using namespace Aws::Utils::Json;

struct JsonShapeSerializer::Impl {
JsonValue m_root;

struct StackEntry {
JsonValue object;
const char* fieldName = nullptr;
bool isList = false;
bool isMap = false;
Aws::Vector<JsonValue> listItems;
};

Aws::Vector<StackEntry> m_stack;

@sbiscigl sbiscigl Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK i see the grand picture of this and this matches our performance profile issues. This is actually much more simple than you are making it, lets start with what we are trying to do:

  • we are prioritizing code gen to avoid a DOM/in memory tree representation of the object like cjson does
  • we are allocating all of the memory up front (or at least try to) so that that we're only allocating and deallocating memory once.

So to start JsonValue will NOT be used in this class. JsonValue calls cjson which will have a multiple heap allocations as it traverses the tree. so that is out the window.

We are going to make it simple we have a character buffer(your calling it a stack in this case), and we just write to it, nothing more.

The basic structure of this would look like

#include <vector>

class json_parser_no_dom {
public:
    json_parser_no_dom(size_t buffer_size) {
        buffer_.reserve(buffer_size);
    }
    json_parser_no_dom(const json_parser_no_dom &other) = delete;
    json_parser_no_dom(json_parser_no_dom &&other) noexcept = default;
    json_parser_no_dom & operator=(const json_parser_no_dom &other) = delete;
    json_parser_no_dom & operator=(json_parser_no_dom &&other) noexcept = default;

    //...
    std::string serialize() { return buffer_; }
private:
    std::string buffer_{};
};

where in this case the parser has a pre-allocated buffer to write stuff into. now we just put stuff into it.

class json_parser_no_dom {
public:
    void write_string(const std::string& str, const std::string& value) {
        buffer_.append("\"");
        buffer_.append(str);
        buffer_.append("\":");
        buffer_.append("\"");
        buffer_.append(value);
        buffer_.append("\"");
    }
};

where you could do something like

auto main() -> int {
    json_parser_no_dom json_parser{100};
    json_parser.begin_structure();
    json_parser.write_string("foo", "bar");
    json_parser.end_structure();
    std::cout << json_parser.serialize() << std::endl;
    return 0;
}

and see

{"foo":"bar"}

so at a very high level I think the problem here is that you are relying on JsonValue and cjson to do the hard parts, and that actually means you are performing heap allocations. what you want to do is rely on the schema for tree walking, and then you just dump the tree to a buffer.

Aws::String m_currentMapKey;

JsonValue& CurrentObject() {
if (m_stack.empty()) return m_root;
return m_stack.back().object;
}

const char* CurrentKey(const Schema& schema) {
if (!m_stack.empty() && m_stack.back().isMap) return m_currentMapKey.c_str();
return schema.GetMemberName();
}

void BeginStructure(const Schema&) {}

@sbiscigl sbiscigl Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whats goin on here, shouldnt we be adding something to the stack when we begin/end a structure?

void EndStructure() {}

void WriteBoolean(const Schema& schema, bool value) {
if (!m_stack.empty() && m_stack.back().isList) {
JsonValue v;

@sbiscigl sbiscigl Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i dont think we wanna declare a JsonValue value here because thats gunna make a heap allocation in cjson

v.AsBool(value);
m_stack.back().listItems.push_back(std::move(v));
} else {
CurrentObject().WithBool(CurrentKey(schema), value);
}
}

void WriteInteger(const Schema& schema, int value) {
if (!m_stack.empty() && m_stack.back().isList) {
JsonValue v;
v.AsInteger(value);
m_stack.back().listItems.push_back(std::move(v));
} else {
CurrentObject().WithInteger(CurrentKey(schema), value);
}
}

void WriteLong(const Schema& schema, int64_t value) {
if (!m_stack.empty() && m_stack.back().isList) {
JsonValue v;
v.AsInt64(value);
m_stack.back().listItems.push_back(std::move(v));
} else {
CurrentObject().WithInt64(CurrentKey(schema), value);
}
}

void WriteDouble(const Schema& schema, double value) {
if (!m_stack.empty() && m_stack.back().isList) {
JsonValue v;
v.AsDouble(value);
m_stack.back().listItems.push_back(std::move(v));
} else {
CurrentObject().WithDouble(CurrentKey(schema), value);
}
}

void WriteString(const Schema& schema, const Aws::String& value) {
if (!m_stack.empty() && m_stack.back().isList) {
JsonValue v;
v.AsString(value);
m_stack.back().listItems.push_back(std::move(v));
} else {
CurrentObject().WithString(CurrentKey(schema), value);
}
}

void WriteTimestamp(const Schema& schema, const DateTime& value) {
if (!m_stack.empty() && m_stack.back().isList) {
JsonValue v;
v.AsDouble(value.SecondsWithMSPrecision());
m_stack.back().listItems.push_back(std::move(v));
} else {
CurrentObject().WithDouble(CurrentKey(schema), value.SecondsWithMSPrecision());
}
}

void WriteBlob(const Schema& schema, const ByteBuffer& value) {
Aws::String encoded = HashingUtils::Base64Encode(value);
if (!m_stack.empty() && m_stack.back().isList) {
JsonValue v;
v.AsString(encoded);
m_stack.back().listItems.push_back(std::move(v));
} else {
CurrentObject().WithString(CurrentKey(schema), encoded);
}
}

void WriteEnum(const Schema& schema, int value) { WriteInteger(schema, value); }

void WriteNull(const Schema& schema) {
if (!m_stack.empty() && m_stack.back().isList) {
JsonValue v;
v.AsNull();
m_stack.back().listItems.push_back(std::move(v));
} else {
CurrentObject().WithNull(CurrentKey(schema));
}
}

void BeginList(const Schema& schema, size_t) {
StackEntry entry;
entry.fieldName = CurrentKey(schema);
entry.isList = true;
m_stack.push_back(std::move(entry));
}

void EndList() {
auto entry = std::move(m_stack.back());
m_stack.pop_back();

Array<JsonValue> arr(entry.listItems.size());
for (size_t i = 0; i < entry.listItems.size(); ++i) {
arr[i] = std::move(entry.listItems[i]);
}

if (!m_stack.empty() && m_stack.back().isList) {
JsonValue listVal;
listVal.WithArray("", std::move(arr));
m_stack.back().listItems.push_back(std::move(listVal));
} else {
CurrentObject().WithArray(entry.fieldName, std::move(arr));
}
}

void BeginMap(const Schema& schema, size_t) {
StackEntry entry;
entry.fieldName = CurrentKey(schema);
entry.isMap = true;
m_stack.push_back(std::move(entry));
}

void WriteMapKey(const Aws::String& key) { m_currentMapKey = key; }

void EndMap() {
auto entry = std::move(m_stack.back());
m_stack.pop_back();

if (!m_stack.empty() && m_stack.back().isList) {
m_stack.back().listItems.push_back(std::move(entry.object));
} else {
CurrentObject().WithObject(entry.fieldName, std::move(entry.object));
}
}

void BeginNestedStructure(const Schema& schema) {
StackEntry entry;
entry.fieldName = CurrentKey(schema);
m_stack.push_back(std::move(entry));
}

void EndNestedStructure() {
auto entry = std::move(m_stack.back());
m_stack.pop_back();

if (!m_stack.empty() && m_stack.back().isList) {
m_stack.back().listItems.push_back(std::move(entry.object));
} else {
CurrentObject().WithObject(entry.fieldName, std::move(entry.object));
}
}

Aws::String GetPayload() const { return m_root.View().WriteCompact(); }
};

JsonShapeSerializer::JsonShapeSerializer() : m_impl(new Impl) {}
JsonShapeSerializer::~JsonShapeSerializer() = default;

void JsonShapeSerializer::BeginStructure(const Schema& schema) { m_impl->BeginStructure(schema); }
void JsonShapeSerializer::EndStructure() { m_impl->EndStructure(); }
void JsonShapeSerializer::WriteBoolean(const Schema& schema, bool value) { m_impl->WriteBoolean(schema, value); }
void JsonShapeSerializer::WriteInteger(const Schema& schema, int value) { m_impl->WriteInteger(schema, value); }
void JsonShapeSerializer::WriteLong(const Schema& schema, int64_t value) { m_impl->WriteLong(schema, value); }
void JsonShapeSerializer::WriteDouble(const Schema& schema, double value) { m_impl->WriteDouble(schema, value); }
void JsonShapeSerializer::WriteString(const Schema& schema, const Aws::String& value) { m_impl->WriteString(schema, value); }
void JsonShapeSerializer::WriteTimestamp(const Schema& schema, const DateTime& value) { m_impl->WriteTimestamp(schema, value); }
void JsonShapeSerializer::WriteBlob(const Schema& schema, const ByteBuffer& value) { m_impl->WriteBlob(schema, value); }
void JsonShapeSerializer::WriteEnum(const Schema& schema, int value) { m_impl->WriteEnum(schema, value); }
void JsonShapeSerializer::WriteNull(const Schema& schema) { m_impl->WriteNull(schema); }
void JsonShapeSerializer::BeginList(const Schema& schema, size_t count) { m_impl->BeginList(schema, count); }
void JsonShapeSerializer::EndList() { m_impl->EndList(); }
void JsonShapeSerializer::BeginMap(const Schema& schema, size_t count) { m_impl->BeginMap(schema, count); }
void JsonShapeSerializer::WriteMapKey(const Aws::String& key) { m_impl->WriteMapKey(key); }
void JsonShapeSerializer::EndMap() { m_impl->EndMap(); }
void JsonShapeSerializer::BeginNestedStructure(const Schema& schema) { m_impl->BeginNestedStructure(schema); }
void JsonShapeSerializer::EndNestedStructure() { m_impl->EndNestedStructure(); }
Aws::String JsonShapeSerializer::GetPayload() const { return m_impl->GetPayload(); }
1 change: 1 addition & 0 deletions tests/aws-cpp-sdk-core-tests/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ file(GLOB UTILS_COMPONENT_REGISTRY_SRC "${CMAKE_CURRENT_SOURCE_DIR}/utils/compon
file(GLOB MONITORING_SRC "${CMAKE_CURRENT_SOURCE_DIR}/monitoring/*.cpp")
file(GLOB SMITHY_TRACING_SRC "${CMAKE_CURRENT_SOURCE_DIR}/smithy/tracing/*.cpp")
file(GLOB SMITHY_CLIENT_SRC "${CMAKE_CURRENT_SOURCE_DIR}/smithy/client/*.cpp")
file(GLOB SMITHY_CLIENT_SRC "${CMAKE_CURRENT_SOURCE_DIR}/smithy/client/schema/*.cpp")
file(GLOB SMITHY_CLIENT_SERIALIZER_SRC "${CMAKE_CURRENT_SOURCE_DIR}/smithy/client/serializer/*.cpp")
file(GLOB SMITHY_CLIENT_FEATURE_SRC "${CMAKE_CURRENT_SOURCE_DIR}/smithy/client/feature/*.cpp")
file(GLOB ENDPOINT_SRC "${CMAKE_CURRENT_SOURCE_DIR}/endpoint/*.cpp")
Expand Down
Loading
Loading