如何在Redis中索引复杂的嵌套JSON以进行查询

时间:2017-11-02 20:41:15

标签: json amazon-ec2 redis aws-cli

我需要一些帮助来理解如何在REDIS中存储和索引以下JSON的所有元素。我的目标是在REDIS中为所有EC2实例存储所有AWS DESCRIBE_INSTACE CLI数据,并查询索引数据以返回具有特定匹配字符串的所有EC2实例。

对于EG - 我需要能够找到并返回属于特定SUBNET / VPC的所有EC2实例

  

可以在REDIS中完成吗?

"Reservations": [
        {
            "Instances": [
                {
                    "Monitoring": {
                        "State": "disabled"
                    },
                    "PublicDnsName": "",
                    "State": {
                        "Code": 16,
                        "Name": "running"
                    },
                    "EbsOptimized": "false",
                    "LaunchTime": "xxxxxxxxx",
                    "PrivateIpAddress": "x.x.x.x",
                    "ProductCodes": [],
                    "VpcId": "xxxxx",
                    "StateTransitionReason": "",
                    "InstanceId": "i-xxxxxxx",
                    "EnaSupport": "true",
                    "ImageId": "ami-xxxxx",
                    "PrivateDnsName": "ip-xxxxxx.ec2.internal",
                    "KeyName": "xxxxxxv",
                    "SecurityGroups": [
                        {
                            "GroupName": "xxx",
                            "GroupId": "sg-xxxx"
                        },
                        {
                            "GroupName": "xxxxxx",
                            "GroupId": "sg-xxxxx"
                        },
                        {
                            "GroupName": "xxxxx",
                            "GroupId": "sg-xxxxxx"
                        },
                        {
                            "GroupName": "xxxxx",
                            "GroupId": "sg-xxxxxx"
                        }
                    ],
                    "ClientToken": "xxxxx",
                    "SubnetId": "subnet-xxxxx",
                    "InstanceType": "t2.micro",
                    "NetworkInterfaces": [
                        {
                            "Status": "in-use",
                            "MacAddress": "xxxxxxxx",
                            "SourceDestCheck": "true",
                            "VpcId": "vpc-xxxxx",
                            "Description": "",
                            "NetworkInterfaceId": "eni-xxxxx",
                            "PrivateIpAddresses": [
                                {
                                    "PrivateDnsName": "ip-xx-ec2.internal",
                                    "Primary": "true",
                                    "PrivateIpAddress": "xxxxx"
                                }
                            ],
                            "PrivateDnsName": "ip-xxxx-xx.ec2.internal",
                            "Attachment": {
                                "Status": "attached",
                                "DeviceIndex": 0,
                                "DeleteOnTermination": "true",
                                "AttachmentId": "eni-attach-xxxxx",
                                "AttachTime": "2017-0xxxxx"
                            },
                            "Groups": [
                                {
                                    "GroupName": "xx",
                                    "GroupId": "sg-xxxx"
                                },
                                {
                                    "GroupName": "xxxx",
                                    "GroupId": "sg-xxx"
                                },
                                {
                                    "GroupName": "xxxx",
                                    "GroupId": "sg-xxx"
                                },
                                {
                                    "GroupName": "xxxx",
                                    "GroupId": "sg-xxxx"
                                }
                            ],
                            "Ipv6Addresses": [],
                            "OwnerId": "xxx",
                            "SubnetId": "subnet-xxxx",
                            "PrivateIpAddress": "1xxxx"
                        }
                    ],
                    "SourceDestCheck": "true",
                    "Placement": {
                        "Tenancy": "default",
                        "GroupName": "",
                        "AvailabilityZone": "us-xxxxxxx"
                    },
                    "Hypervisor": "xen",
                    "BlockDeviceMappings": [
                        {
                            "DeviceName": "/dev/xxxxxx",
                            "Ebs": {
                                "Status": "attached",
                                "DeleteOnTermination": "true",
                                "VolumeId": "vol-xxxxxx",
                                "AttachTime": "2017-xxxxxxx"
                            }
                        }
                    ],
                    "Architecture": "x86_64",
                    "RootDeviceType": "ebs",
                    "IamInstanceProfile": {
                        "Id": "xxxxxxxx",
                        "Arn": "arn:aws:iam::xxxxxxx"
                    },
                    "RootDeviceName": "/dev/xxxxx",
                    "VirtualizationType": "hvm",
                    "Tags": [
                        {
                            "Value": "xxxxxx",
                            "Key": "aws:cloudformation:stack-name"
                        },
                        {
                            "Value": "xxxxxxx",
                            "Key": "aws:cloudformation:logical-id"
                        },
                        {
                            "Value": "arn:aws:cloudformation:xxxxxx",
                            "Key": "aws:cloudformation:stack-id"
                        }
                    ],
                    "AmiLaunchIndex": 0
                }
            ],
            "ReservationId": "r-xxxxx",
            "RequesterId": "xxxxx",
            "Groups": [],
            "OwnerId": "xxxxxx"
        }
    ]  
}

1 个答案:

答案 0 :(得分:0)

是的,这是可以做到的。然而,重要的是要意识到 Redis 本身并不真正关心 JSON,因此您必须将数据转换为可以有效查询的格式。这种“规范化”(让我们暂时称之为)必须在应用程序级别发生。换句话说 - 你必须处理这个问题,而不是 Redis。

<块引用>

对于 EG - 我需要能够找到并返回属于特定 SUBNET/VPC 的所有 EC2 实例

让我们看一下这个例子。对此数据建模的一种方法可能如下所示:

  1. 为每个实例分配一个唯一的 ID。
  2. 将每个实例存储为 instance:{instance_id} 下的 JSON 编码 blob。
  3. 将实例 ID 存储在其各自的 instance_by_subnet:subnet-xxxxx 集下。

更具体地说 - 使用您提供的示例 - 这可能如下所示:

SET instance:1 {"Monitoring":.......} # Store first instance
SADD instance_by_subnet:subnet-abc123 1 # Add instance id to relevant set

通过这种方式,您可以通过调用 SMEMBERS instance_by_subnet:subnet-abc123 来确定实例属于特定子集。您必须发出另一个 GET 命令才能实际读取 JSON blob。

有多种方法可以维护这些索引,还有许多其他数据结构在建模数据时会派上用场。如果您有兴趣,我写了 an article 更详细地介绍了这一点。