scrapi.util.json_without_bytes

Here are the examples of the python api scrapi.util.json_without_bytes taken from open source projects. By voting up you can indicate which examples are most useful and appropriate.

3 Examples 7

Example 1

Project: scrapi Source File: document.py
Function: init
    def __init__(self, attributes, validate=True, clean=False):
        ''' Initializes a docuement

            :param dict attributes: the dictionary representation of a docuement
            :param bool validate: If true, the object will be validated before creation
            :param bool clean: If true, optional fields that are null will be deleted
        '''
        attributes = attributes or {}
        # validate a version of the attributes that are safe to check
        # against the JSON schema

        # Allows validation in python3
        self.attributes = json_without_bytes(copy.deepcopy(attributes))
        if clean:
            self.attributes = strip_empty(self.attributes, required=self.schema.get('required', []))
        if validate:
            self.validate()

Example 2

Project: scrapi Source File: postgres.py
Function: create
    def create(self, attributes):
        attributes = json_without_bytes(attributes)
        Docuement.objects.create(
            source=attributes['source'],
            docID=attributes['docID'],
            providerUpdatedDateTime=None,
            raw=attributes,
            normalized=None
        ).save()

Example 3

Project: scrapi Source File: storage.py
Function: write
    def write(self, source, doc_id, filename, content):
        filepath = 'archive/{}/{}/{}.json'.format(source, doc_id, filename)

        if not os.path.exists(os.path.dirname(filepath)):
            os.makedirs(os.path.dirname(filepath))

        with open(filepath, 'w') as f:
            f.write(json.dumps(json_without_bytes(content), indent=4))