OTE-API Core Strategies¶
This page provides documentation for the oteapi.strategies
submodule, where all the core OTE-API strategies are located.
These strategies will always be available when setting up a server based on the OTE-API Core package.
download
¶
file
¶
Download strategy class for the file
scheme.
DownloadFileContent
¶
Bases: AttrDict
Class for returning values from Download File strategy.
Source code in oteapi/strategies/download/file.py
59 60 61 62 |
|
key: str = Field(..., description='Key to access the data in the cache.')
class-attribute
instance-attribute
¶
FileConfig
¶
Bases: AttrDict
File-specific Configuration Data Model.
Source code in oteapi/strategies/download/file.py
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
|
datacache_config: Optional[DataCacheConfig] = Field(None, description='Configurations for the data cache for storing the downloaded file content.')
class-attribute
instance-attribute
¶
encoding: Optional[str] = Field(None, description='Encoding used when opening the file. The default is platform dependent.')
class-attribute
instance-attribute
¶
text: bool = Field(False, description='Whether the file should be opened in text mode. If `False`, the file will be opened in bytes mode.')
class-attribute
instance-attribute
¶
FileResourceConfig
¶
Bases: ResourceConfig
File download strategy filter config.
Source code in oteapi/strategies/download/file.py
40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
|
configuration: FileConfig = Field(FileConfig(), description='File download strategy-specific configuration.')
class-attribute
instance-attribute
¶
downloadUrl: FileUrl = Field(..., description='The file URL, which will be downloaded.')
class-attribute
instance-attribute
¶
ensure_path_exists(value)
classmethod
¶
Ensure path
is defined in downloadUrl
.
Source code in oteapi/strategies/download/file.py
50 51 52 53 54 55 56 |
|
FileStrategy
¶
Strategy for retrieving data from a local file.
Registers strategies:
("scheme", "file")
Source code in oteapi/strategies/download/file.py
65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 |
|
download_config: FileResourceConfig
instance-attribute
¶
get()
¶
Read local file.
Source code in oteapi/strategies/download/file.py
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 |
|
initialize()
¶
Initialize.
Source code in oteapi/strategies/download/file.py
77 78 79 |
|
https
¶
Download strategy class for http/https
HTTPDownloadContent
¶
Bases: AttrDict
Class for returning values from Download HTTPS strategy.
Source code in oteapi/strategies/download/https.py
110 111 112 113 |
|
key: str = Field(..., description='Key to access the data in the cache.')
class-attribute
instance-attribute
¶
HTTPSConfig
¶
Bases: AttrDict
HTTP(S)-specific Configuration Data Model.
Source code in oteapi/strategies/download/https.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 |
|
cookies: Optional[dict[str, str]] = Field(None, description='Cookies to be included in the download request.')
class-attribute
instance-attribute
¶
datacache_config: Optional[DataCacheConfig] = Field(None, description='Configurations for the data cache for storing the downloaded file content.')
class-attribute
instance-attribute
¶
headers: Optional[dict[str, str]] = Field(None, description='HTTP headers to be included in the download request.')
class-attribute
instance-attribute
¶
http_method: Literal['GET', 'POST'] = Field('GET', description='HTTP method to use for the download request. Only GET and POST are supported.')
class-attribute
instance-attribute
¶
post_body: Optional[Union[dict[str, Any], list[tuple[str, Any]], bytes]] = Field(None, description='The body of the POST request. This can be a a dictionary, list of tuples or bytes. This field is mutually exclusive with `post_body_json`.')
class-attribute
instance-attribute
¶
post_body_json: Optional[Any] = Field(None, description='The body of the POST request as a JSON serializable Python object. This will be serialized to JSON and sent as the body of the POST request. This field is mutually exclusive with `post_body`.')
class-attribute
instance-attribute
¶
query_parameters: Optional[dict[str, Union[str, list[str]]]] = Field(None, description='Query parameters to be included in the download request. Note, these can be included directly in the `downloadURL` as well.')
class-attribute
instance-attribute
¶
HTTPSResourceConfig
¶
Bases: ResourceConfig
HTTP(S) download strategy filter config.
Source code in oteapi/strategies/download/https.py
99 100 101 102 103 104 105 106 107 |
|
HTTPSStrategy
¶
Strategy for retrieving data via http.
Registers strategies:
("scheme", "http")
("scheme", "https")
Source code in oteapi/strategies/download/https.py
116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 |
|
download_config: HTTPSResourceConfig
instance-attribute
¶
get()
¶
Download via http/https and store on local cache.
Source code in oteapi/strategies/download/https.py
133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 |
|
initialize()
¶
Initialize.
Source code in oteapi/strategies/download/https.py
129 130 131 |
|
sftp
¶
Strategy class for sftp/ftp
AnyFtpUrl = Annotated[Url, UrlConstraints(allowed_schemes=['ftp', 'sftp'])]
module-attribute
¶
SFTPConfig
¶
Bases: AttrDict
(S)FTP-specific Configuration Data Model.
Source code in oteapi/strategies/download/sftp.py
20 21 22 23 24 25 26 27 28 29 |
|
datacache_config: Optional[DataCacheConfig] = Field(None, description='Configurations for the data cache for storing the downloaded file content.')
class-attribute
instance-attribute
¶
SFTPContent
¶
Bases: AttrDict
Class for returning values from Download SFTP strategy.
Source code in oteapi/strategies/download/sftp.py
43 44 45 46 |
|
key: str = Field(..., description='Key to access the data in the cache.')
class-attribute
instance-attribute
¶
SFTPResourceConfig
¶
Bases: ResourceConfig
(S)FTP download strategy filter config.
Source code in oteapi/strategies/download/sftp.py
32 33 34 35 36 37 38 39 40 |
|
SFTPStrategy
¶
Strategy for retrieving data via sftp.
Registers strategies:
("scheme", "ftp")
("scheme", "sftp")
Source code in oteapi/strategies/download/sftp.py
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 |
|
download_config: SFTPResourceConfig
instance-attribute
¶
get()
¶
Download via sftp
Source code in oteapi/strategies/download/sftp.py
66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 |
|
initialize()
¶
Initialize.
Source code in oteapi/strategies/download/sftp.py
62 63 64 |
|
filter
¶
crop_filter
¶
Demo-filter strategy
CropFilterContent
¶
Bases: AttrDict
Return model for CropImageFilter
.
Source code in oteapi/strategies/filter/crop_filter.py
39 40 41 42 43 44 |
|
imagecrop: tuple[int, int, int, int] = Field(..., description='Box cropping parameters (left, top, right, bottom).')
class-attribute
instance-attribute
¶
CropImageConfig
¶
Bases: AttrDict
Configuration model for crop data.
Source code in oteapi/strategies/filter/crop_filter.py
19 20 21 22 23 24 |
|
crop: Optional[tuple[int, int, int, int]] = Field(None, description='Box cropping parameters (left, top, right, bottom).')
class-attribute
instance-attribute
¶
CropImageFilter
¶
Strategy for cropping an image.
Registers strategies:
("filterType", "filter/crop")
Source code in oteapi/strategies/filter/crop_filter.py
47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
|
filter_config: CropImageFilterConfig
instance-attribute
¶
get()
¶
Execute strategy and return a dictionary
Source code in oteapi/strategies/filter/crop_filter.py
68 69 70 |
|
initialize()
¶
Initialize strategy and return a dictionary.
Source code in oteapi/strategies/filter/crop_filter.py
59 60 61 62 63 64 65 66 |
|
CropImageFilterConfig
¶
Bases: FilterConfig
Crop filter strategy filter config.
Source code in oteapi/strategies/filter/crop_filter.py
27 28 29 30 31 32 33 34 35 36 |
|
configuration: CropImageConfig = Field(..., description='Image crop filter strategy-specific configuration.')
class-attribute
instance-attribute
¶
filterType: Literal['filter/crop'] = Field('filter/crop', description=FilterConfig.model_fields['filterType'].description)
class-attribute
instance-attribute
¶
sql_query_filter
¶
SQL query filter strategy.
SQLQueryFilter
¶
Strategy for a SQL query filter.
Registers strategies:
("filterType", "filter/sql")
Source code in oteapi/strategies/filter/sql_query_filter.py
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
|
filter_config: SqlQueryFilterConfig
instance-attribute
¶
get()
¶
Execute strategy and return a dictionary.
Source code in oteapi/strategies/filter/sql_query_filter.py
50 51 52 |
|
initialize()
¶
Initialize strategy.
Source code in oteapi/strategies/filter/sql_query_filter.py
46 47 48 |
|
SqlQueryContent
¶
Bases: AttrDict
Class for returning values from SQL Query data model.
Source code in oteapi/strategies/filter/sql_query_filter.py
28 29 30 31 |
|
sqlquery: str = Field(..., description='A SQL query string.')
class-attribute
instance-attribute
¶
SqlQueryFilterConfig
¶
Bases: FilterConfig
SQL query filter strategy filter config.
Source code in oteapi/strategies/filter/sql_query_filter.py
18 19 20 21 22 23 24 25 |
|
mapping
¶
mapping
¶
Mapping filter strategy.
MappingStrategy
¶
Strategy for a mapping.
The mapping strategy simply adds more prefixes and triples to the
prefixes
and triples
fields in the session such that they are
available for other strategies, like function strategies that convert
between data models.
Nothing is returned to avoid deleting existing mappings.
Registers strategies:
("mappingType", "triples")
Source code in oteapi/strategies/mapping/mapping.py
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
|
mapping_config: MappingConfig
instance-attribute
¶
get()
¶
Execute strategy and return a dictionary.
Source code in oteapi/strategies/mapping/mapping.py
52 53 54 |
|
initialize()
¶
Initialize strategy.
Source code in oteapi/strategies/mapping/mapping.py
45 46 47 48 49 50 |
|
MappingStrategyConfig
¶
Bases: AttrDict
AttrDict model for mappings.
Source code in oteapi/strategies/mapping/mapping.py
10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
|
prefixes: dict[str, str] = Field(..., description='Dictionary of shortnames that expands to an IRI given as local value/IRI-expansion-pairs.')
class-attribute
instance-attribute
¶
triples: list[RDFTriple] = Field(..., description='List of semantic triples given as (subject, predicate, object).')
class-attribute
instance-attribute
¶
parse
¶
application_json
¶
Strategy class for application/json.
JSONConfig
¶
Bases: AttrDict
JSON parse-specific Configuration Data Model.
Source code in oteapi/strategies/parse/application_json.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
|
datacache_config: Optional[DataCacheConfig] = Field(None, description='Configurations for the data cache for storing the downloaded file content.')
class-attribute
instance-attribute
¶
downloadUrl: Optional[HostlessAnyUrl] = Field(None, description='The HTTP(S) URL, which will be downloaded.')
class-attribute
instance-attribute
¶
mediaType: Literal['application/json'] = Field('application/json', description='The media type')
class-attribute
instance-attribute
¶
JSONDataParseStrategy
¶
Parse strategy for JSON.
Registers strategies:
("parserType", "parser/json")
Source code in oteapi/strategies/parse/application_json.py
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 |
|
parse_config: JSONParserConfig
instance-attribute
¶
get()
¶
Parse json.
Source code in oteapi/strategies/parse/application_json.py
75 76 77 78 79 80 81 82 83 84 85 86 |
|
initialize()
¶
Initialize.
Source code in oteapi/strategies/parse/application_json.py
71 72 73 |
|
JSONParseContent
¶
Bases: AttrDict
Class for returning values from JSON Parse.
Source code in oteapi/strategies/parse/application_json.py
53 54 55 56 |
|
content: dict = Field(..., description='Content of the JSON document.')
class-attribute
instance-attribute
¶
JSONParserConfig
¶
Bases: ParserConfig
JSON parse strategy filter config.
Source code in oteapi/strategies/parse/application_json.py
41 42 43 44 45 46 47 48 49 50 |
|
application_vnd_sqlite
¶
Strategy class for application/vnd.sqlite3.
SqLiteParseContent
¶
Bases: AttrDict
Configuration model for SqLiteParse.
Source code in oteapi/strategies/parse/application_vnd_sqlite.py
82 83 84 85 |
|
result: list = Field(..., description='List of results from the query.')
class-attribute
instance-attribute
¶
SqliteConfig
¶
Bases: AttrDict
Configuration data model for
SqliteParseStrategy
.
Source code in oteapi/strategies/parse/application_vnd_sqlite.py
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
|
datacache_config: Optional[DataCacheConfig] = Field(None, description='Configuration options for the local data cache.')
class-attribute
instance-attribute
¶
downloadUrl: Optional[HostlessAnyUrl] = Field(None, description=ResourceConfig.model_fields['downloadUrl'].description)
class-attribute
instance-attribute
¶
mediaType: Literal['application/vnd.sqlite3'] = Field('application/vnd.sqlite3', description=ResourceConfig.model_fields['mediaType'].description)
class-attribute
instance-attribute
¶
sqlquery: str = Field('', description='A SQL query string.')
class-attribute
instance-attribute
¶
SqliteParseStrategy
¶
Parse strategy for SQLite.
Purpose of this strategy: Download a SQLite database using downloadUrl
and run a
SQL query on the database to return all relevant rows.
Source code in oteapi/strategies/parse/application_vnd_sqlite.py
88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 |
|
parse_config: SqliteParserConfig
instance-attribute
¶
get()
¶
Parse SQLite query responses.
Source code in oteapi/strategies/parse/application_vnd_sqlite.py
103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 |
|
initialize()
¶
Initialize strategy.
Source code in oteapi/strategies/parse/application_vnd_sqlite.py
99 100 101 |
|
SqliteParserConfig
¶
Bases: ParserConfig
SQLite parse strategy resource config.
Source code in oteapi/strategies/parse/application_vnd_sqlite.py
51 52 53 54 55 56 57 58 59 60 |
|
configuration: SqliteConfig = Field(..., description='SQLite parse strategy-specific configuration.')
class-attribute
instance-attribute
¶
parserType: Literal['parser/sqlite3'] = Field('parser/sqlite3', description=ParserConfig.model_fields['parserType'].description)
class-attribute
instance-attribute
¶
create_connection(db_file)
¶
Create a database connection to SQLite database.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
db_file |
Path
|
Full path to SQLite database file. |
required |
Raises:
Type | Description |
---|---|
Error
|
If a DB connection cannot be made. |
Returns:
Type | Description |
---|---|
Connection
|
Connection object. |
Source code in oteapi/strategies/parse/application_vnd_sqlite.py
63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 |
|
excel_xlsx
¶
Strategy class for workbook/xlsx.
XLSXParseConfig
¶
Bases: AttrDict
Data model for retrieving a rectangular section of an Excel sheet.
Source code in oteapi/strategies/parse/excel_xlsx.py
39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 |
|
col_from: Optional[Union[int, str]] = Field(None, description='Excel column number or label of first column. Defaults to first assigned column.')
class-attribute
instance-attribute
¶
col_to: Optional[Union[int, str]] = Field(None, description='Excel column number or label of last column. Defaults to last assigned column.')
class-attribute
instance-attribute
¶
datacache_config: Optional[DataCacheConfig] = Field(None, description='Configurations for the data cache for retrieving the downloaded file content.')
class-attribute
instance-attribute
¶
downloadUrl: Optional[HostlessAnyUrl] = Field(None, description=ResourceConfig.model_fields['downloadUrl'].description)
class-attribute
instance-attribute
¶
download_config: AttrDict = Field(AttrDict(), description='Configurations provided to a download strategy.')
class-attribute
instance-attribute
¶
header: Optional[list[str]] = Field(None, description='Optional list of column names, specifying the columns to return. These names they should match cells in `header_row`.')
class-attribute
instance-attribute
¶
header_row: Optional[int] = Field(None, description='Row number with the headers. Defaults to `1` if header is given, otherwise `None`.')
class-attribute
instance-attribute
¶
mediaType: Literal['application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'] = Field('application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', description=ResourceConfig.model_fields['mediaType'].description)
class-attribute
instance-attribute
¶
new_header: Optional[list[str]] = Field(None, description='Optional list of new column names replacing `header` in the output.')
class-attribute
instance-attribute
¶
row_from: Optional[int] = Field(None, description='Excel row number of first row. Defaults to first assigned row.')
class-attribute
instance-attribute
¶
row_to: Optional[int] = Field(None, description='Excel row number of last row. Defaults to last assigned row.')
class-attribute
instance-attribute
¶
worksheet: str = Field(..., description='Name of worksheet to load.')
class-attribute
instance-attribute
¶
XLSXParseContent
¶
Bases: AttrDict
Class for returning values from XLSXParse.
Source code in oteapi/strategies/parse/excel_xlsx.py
30 31 32 33 34 35 36 |
|
data: dict[str, list] = Field(..., description='A dict with column-name/column-value pairs. The values are lists.')
class-attribute
instance-attribute
¶
XLSXParseParserConfig
¶
Bases: ParserConfig
XLSX parse strategy resource config.
Source code in oteapi/strategies/parse/excel_xlsx.py
109 110 111 112 113 114 115 116 117 118 |
|
configuration: XLSXParseConfig = Field(..., description='SQLite parse strategy-specific configuration.')
class-attribute
instance-attribute
¶
parserType: Literal['parser/excel_xlsx'] = Field('parser/excel_xlsx', description=ParserConfig.model_fields['parserType'].description)
class-attribute
instance-attribute
¶
XLSXParseStrategy
¶
Parse strategy for Excel XLSX files.
Registers strategies:
("parserType", "excel_xlsx")
Source code in oteapi/strategies/parse/excel_xlsx.py
176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 |
|
parse_config: XLSXParseParserConfig
instance-attribute
¶
get()
¶
Parses selected region of an excel file.
Returns:
Type | Description |
---|---|
XLSXParseContent
|
A dict with column-name/column-value pairs. The values are lists. |
Source code in oteapi/strategies/parse/excel_xlsx.py
192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 |
|
initialize()
¶
Initialize.
Source code in oteapi/strategies/parse/excel_xlsx.py
188 189 190 |
|
get_column_indices(model, worksheet)
¶
Helper function returning a list of column indices.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
model |
XLSXParseConfig
|
The parsed data model. |
required |
worksheet |
Worksheet
|
Excel worksheet, from which the header values will be retrieved. |
required |
Returns:
Type | Description |
---|---|
Iterable[int]
|
A list of column indices. |
Source code in oteapi/strategies/parse/excel_xlsx.py
153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 |
|
set_model_defaults(model, worksheet)
¶
Update data model model
with default values obtained from worksheet
.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
model |
XLSXParseConfig
|
The parsed data model. |
required |
worksheet |
Worksheet
|
Excel worksheet, from which the default values will be obtained. |
required |
Source code in oteapi/strategies/parse/excel_xlsx.py
121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 |
|
image
¶
Strategy class for image/jpg.
ImageConfig
¶
Bases: AttrDict
Configuration data model for
ImageDataParseStrategy
.
Source code in oteapi/strategies/parse/image.py
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 |
|
crop: Optional[tuple[int, int, int, int]] = Field(None, description='Box cropping parameters (left, top, right, bottom).', validation_alias=AliasChoices('crop', 'imagecrop'))
class-attribute
instance-attribute
¶
datacache_config: Optional[DataCacheConfig] = Field(None, description='Configuration options for the local data cache.')
class-attribute
instance-attribute
¶
downloadUrl: Optional[HostlessAnyUrl] = Field(None, description=ResourceConfig.model_fields['downloadUrl'].description)
class-attribute
instance-attribute
¶
image_key: Optional[str] = Field(None, description='Key to use when storing the image data in datacache.')
class-attribute
instance-attribute
¶
image_mode: Optional[str] = Field(None, description='Pillow mode to convert image into. See https://pillow.readthedocs.io/en/stable/handbook/concepts.html for details.')
class-attribute
instance-attribute
¶
mediaType: Optional[Literal['image/jpg', 'image/jpeg', 'image/jp2', 'image/png', 'image/gif', 'image/tiff', 'image/eps']] = Field(None, description=ResourceConfig.model_fields['mediaType'].description)
class-attribute
instance-attribute
¶
ImageDataParseStrategy
¶
Parse strategy for images.
This strategy uses Pillow to read a raw image from the data cache, converts it into a NumPy array and stores the new array in the data cache.
It also supports simple cropping and image conversions.
The key to the new array and other metadata is returned. See
ImageParseContent
for more info.
Source code in oteapi/strategies/parse/image.py
133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 |
|
parse_config: ImageParserConfig
instance-attribute
¶
get()
¶
Execute the strategy.
Source code in oteapi/strategies/parse/image.py
155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 |
|
initialize()
¶
Initialize strategy.
Source code in oteapi/strategies/parse/image.py
151 152 153 |
|
ImageParseContent
¶
Bases: AttrDict
Configuration model for the returned content from the Image parser.
See
Pillow handbook
for more details on image_mode
, image_palette
, and image_info
.
Source code in oteapi/strategies/parse/image.py
103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 |
|
image_info: dict = Field({}, description='Additional information about the image.')
class-attribute
instance-attribute
¶
image_key: str = Field(..., description='Key with which the image content is stored in the data cache.')
class-attribute
instance-attribute
¶
image_mode: str = Field(..., description="Image mode. Examples: 'L', 'P', 'RGB', 'RGBA'...")
class-attribute
instance-attribute
¶
image_palette_key: Optional[str] = Field(None, description="Datacache key for colour palette if mode is 'P'.")
class-attribute
instance-attribute
¶
image_size: tuple[int, int] = Field(..., description='Image size (width, height).')
class-attribute
instance-attribute
¶
ImageParserConfig
¶
Bases: ParserConfig
Image parse strategy resource config.
Source code in oteapi/strategies/parse/image.py
78 79 80 81 82 83 84 85 86 87 88 |
|
SupportedFormat
¶
Bases: Enum
Supported formats for ImageDataParseStrategy
.
Source code in oteapi/strategies/parse/image.py
91 92 93 94 95 96 97 98 99 100 |
|
eps = 'EPS'
class-attribute
instance-attribute
¶
gif = 'GIF'
class-attribute
instance-attribute
¶
jp2 = 'JPEG2000'
class-attribute
instance-attribute
¶
jpeg = 'JPEG'
class-attribute
instance-attribute
¶
jpg = 'JPEG'
class-attribute
instance-attribute
¶
png = 'PNG'
class-attribute
instance-attribute
¶
tiff = 'TIFF'
class-attribute
instance-attribute
¶
postgres
¶
Strategy class for application/vnd.postgresql
PostgresConfig
¶
Bases: AttrDict
Configuration data model for
PostgresParserStrategy
.
Source code in oteapi/strategies/parse/postgres.py
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 |
|
accessService: Literal['postgres'] = Field('postgres', description=ResourceConfig.model_fields['accessService'].description)
class-attribute
instance-attribute
¶
accessUrl: Optional[HostlessAnyUrl] = Field(None, description=ResourceConfig.model_fields['accessUrl'].description)
class-attribute
instance-attribute
¶
datacache_config: Optional[DataCacheConfig] = Field(None, description='Configuration options for the local data cache.')
class-attribute
instance-attribute
¶
dbname: Optional[str] = Field(None, description='postgres dbname name')
class-attribute
instance-attribute
¶
password: Optional[str] = Field(None, description='postgres password')
class-attribute
instance-attribute
¶
sqlquery: str = Field('', description='A SQL query string.')
class-attribute
instance-attribute
¶
user: Optional[str] = Field(None, description='postgres server username')
class-attribute
instance-attribute
¶
adjust_url(data)
classmethod
¶
Model Validator Verifies configuration consistency, merge configurations and update the accessUrl property.
Source code in oteapi/strategies/parse/postgres.py
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 |
|
PostgresParserConfig
¶
Bases: ParserConfig
Postgresql parse strategy config
Source code in oteapi/strategies/parse/postgres.py
111 112 113 114 115 116 117 118 119 120 121 122 123 |
|
configuration: PostgresConfig = Field(..., description='Configuration for resource. Values in the accessURL take precedence.')
class-attribute
instance-attribute
¶
parserType: Literal['parser/postgres'] = Field('parser/postgres', description='Type of registered resource strategy.')
class-attribute
instance-attribute
¶
PostgresParserContent
¶
Bases: AttrDict
Configuration model for PostgresParser.
Source code in oteapi/strategies/parse/postgres.py
145 146 147 148 |
|
result: list = Field(..., description='List of results from the query.')
class-attribute
instance-attribute
¶
PostgresParserStrategy
¶
Resource strategy for Postgres.
Purpose of this strategy: Connect to a postgres DB and run a SQL query on the dbname to return all relevant rows.
Source code in oteapi/strategies/parse/postgres.py
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 |
|
parser_config: PostgresParserConfig
instance-attribute
¶
get()
¶
Resource Postgres query responses.
Source code in oteapi/strategies/parse/postgres.py
166 167 168 169 170 171 172 173 174 175 176 |
|
initialize()
¶
Initialize strategy.
Source code in oteapi/strategies/parse/postgres.py
162 163 164 |
|
create_connection(url)
¶
Create a dbname connection to Postgres dbname.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
url |
str
|
A valid PostgreSQL URL. |
required |
Raises:
Type | Description |
---|---|
Error
|
If a DB connection cannot be made. |
Returns:
Type | Description |
---|---|
Connection
|
Connection object. |
Source code in oteapi/strategies/parse/postgres.py
126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 |
|
text_csv
¶
Strategy class for parser/csv.
CSVDialect: type[Enum] = Enum(value='CSVDialect', names={dialect.upper(): dialectfor dialect in csv.list_dialects()}, module=__name__, type=str)
module-attribute
¶
CSV dialects.
All available dialects are retrieved through the csv.list_dialects()
function,
and will thus depend on the currently loaded and used Python interpreter.
CSVConfig
¶
Bases: AttrDict
CSV parse-specific Configuration Data Model.
Source code in oteapi/strategies/parse/text_csv.py
222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 |
|
datacache_config: Optional[DataCacheConfig] = Field(None, description='Configurations for the data cache for storing the downloaded file content.')
class-attribute
instance-attribute
¶
dialect: DialectFormatting = Field(DialectFormatting(), description='Dialect and formatting parameters. See [the Python docs](https://docs.python.org/3/library/csv.html#csv-fmt-params) for more information.')
class-attribute
instance-attribute
¶
downloadUrl: Optional[HostlessAnyUrl] = Field(None, description=ResourceConfig.model_fields['downloadUrl'].description)
class-attribute
instance-attribute
¶
mediaType: Literal['text/csv'] = Field('text/csv', description=ResourceConfig.model_fields['mediaType'].description)
class-attribute
instance-attribute
¶
reader: ReaderConfig = Field(ReaderConfig(), description='CSV DictReader configuration parameters. See [the Python docs](https://docs.python.org/3/library/csv.html#csv.DictReader) for more information.')
class-attribute
instance-attribute
¶
CSVParseContent
¶
Bases: AttrDict
Class for returning values from CSV Parse.
Source code in oteapi/strategies/parse/text_csv.py
273 274 275 276 277 278 |
|
content: dict[Union[str, None], list[Any]] = Field(..., description='Content of the CSV document.')
class-attribute
instance-attribute
¶
CSVParseStrategy
¶
Parse strategy for CSV files.
Source code in oteapi/strategies/parse/text_csv.py
281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 |
|
parse_config: CSVParserConfig
instance-attribute
¶
get()
¶
Parse CSV.
Source code in oteapi/strategies/parse/text_csv.py
291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 |
|
initialize()
¶
Initialize.
Source code in oteapi/strategies/parse/text_csv.py
287 288 289 |
|
CSVParserConfig
¶
Bases: ParserConfig
CSV parse strategy filter config.
Source code in oteapi/strategies/parse/text_csv.py
261 262 263 264 265 266 267 268 269 270 |
|
DialectFormatting
¶
Bases: BaseModel
Dialect and formatting parameters for CSV.
See the Python docs for more information.
Note
As Dialect.lineterminator
is hardcoded in csv.reader
, it is left out of
this model.
Source code in oteapi/strategies/parse/text_csv.py
63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 |
|
base: Optional[CSVDialect] = Field(None, description="A specific CSV dialect, e.g., 'excel'. Any other parameters here will overwrite the preset dialect parameters for the specified dialect.")
class-attribute
instance-attribute
¶
delimiter: Optional[str] = Field(None, description='A one-character string used to separate fields. See [the Python docs entry](https://docs.python.org/3/library/csv.html#csv.Dialect.delimiter) for more information.', min_length=1, max_length=1)
class-attribute
instance-attribute
¶
doublequote: Optional[bool] = Field(None, description='Controls how instances of [`quotechar`][oteapi.strategies.parse.text_csv.DialectFormatting.quotechar] appearing inside a field should themselves be quoted. When `True`, the character is doubled. When `False`, the [`escapechar`][oteapi.strategies.parse.text_csv.DialectFormatting.escapechar] is used as a prefix to the [`quotechar`][oteapi.strategies.parse.text_csv.DialectFormatting.quotechar]. See [the Python docs entry](https://docs.python.org/3/library/csv.html#csv.Dialect.doublequote) for more information.')
class-attribute
instance-attribute
¶
escapechar: Optional[str] = Field(None, description='A one-character string used by the writer to escape the [`delimiter`][oteapi.strategies.parse.text_csv.DialectFormatting.delimiter] if [`quoting`][oteapi.strategies.parse.text_csv.DialectFormatting.quoting] is set to [`QUOTE_NONE`][oteapi.strategies.parse.text_csv.QuoteConstants.QUOTE_NONE] and the [`quotechar`][oteapi.strategies.parse.text_csv.DialectFormatting.quotechar] if [`doublequote`][oteapi.strategies.parse.text_csv.DialectFormatting.doublequote] is `False`. On reading, the [`escapechar`][oteapi.strategies.parse.text_csv.DialectFormatting.escapechar] removes any special meaning from the following character. See [the Python docs entry](https://docs.python.org/3/library/csv.html#csv.Dialect.escapechar) for more information.', min_length=1, max_length=1)
class-attribute
instance-attribute
¶
quotechar: Optional[str] = Field(None, description='A one-character string used to quote fields containing special characters, such as the [`delimiter`][oteapi.strategies.parse.text_csv.DialectFormatting.delimiter] or [`quotechar`][oteapi.strategies.parse.text_csv.DialectFormatting.quotechar], or which contain new-line characters. See [the Python docs entry](https://docs.python.org/3/library/csv.html#csv.Dialect.quotechar) for more information.', min_length=1, max_length=1)
class-attribute
instance-attribute
¶
quoting: Optional[QuoteConstants] = Field(None, description='Controls when quotes should be generated by the writer and recognised by the reader. It can take on any of the `QUOTE_*` constants (see section [Module Contents](https://docs.python.org/3/library/csv.html#csv-contents)). See [the Python docs entry](https://docs.python.org/3/library/csv.html#csv.Dialect.quoting) for more information.')
class-attribute
instance-attribute
¶
skipinitialspace: Optional[bool] = Field(None, description='When `True`, whitespace immediately following the [`delimiter`][oteapi.strategies.parse.text_csv.DialectFormatting.delimiter] is ignored. See [the Python docs entry](https://docs.python.org/3/library/csv.html#csv.Dialect.skipinitialspace) for more information.')
class-attribute
instance-attribute
¶
strict: Optional[bool] = Field(None, description='When `True`, raise exception [Error](https://docs.python.org/3/library/csv.html#csv.Error) on bad CSV input. See [the Python docs entry](https://docs.python.org/3/library/csv.html#csv.Dialect.strict) for more information.')
class-attribute
instance-attribute
¶
validate_dialect_base(value)
classmethod
¶
Ensure the given base
dialect is registered locally.
Source code in oteapi/strategies/parse/text_csv.py
172 173 174 175 176 177 178 179 180 181 |
|
QuoteConstants
¶
Bases: str
, Enum
CSV module QUOTE_*
constants.
Source code in oteapi/strategies/parse/text_csv.py
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
|
QUOTE_ALL = 'QUOTE_ALL'
class-attribute
instance-attribute
¶
QUOTE_MINIMAL = 'QUOTE_MINIMAL'
class-attribute
instance-attribute
¶
QUOTE_NONE = 'QUOTE_NONE'
class-attribute
instance-attribute
¶
QUOTE_NONUMERIC = 'QUOTE_NONNUMERIC'
class-attribute
instance-attribute
¶
csv_constant()
¶
Return the CSV lib equivalent constant.
Source code in oteapi/strategies/parse/text_csv.py
39 40 41 42 43 44 45 46 |
|
ReaderConfig
¶
Bases: BaseModel
CSV DictReader configuration parameters.
See the Python docs for more information.
Source code in oteapi/strategies/parse/text_csv.py
184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 |
|
encoding: str = Field('utf8', description='The file encoding.')
class-attribute
instance-attribute
¶
fieldnames: Optional[list[str]] = Field(None, description='List of headers. If not set, the values in the first row of the CSV file will be used as the field names.')
class-attribute
instance-attribute
¶
restkey: Optional[Hashable] = Field(None, description='If a row has more fields than [`fieldnames`][oteapi.strategies.parse.text_csv.ReaderConfig.fieldnames], the remaining data is put in a list and stored with the field name specified by [`restkey`][oteapi.strategies.parse.text_csv.ReaderConfig.restkey].')
class-attribute
instance-attribute
¶
restval: Optional[Any] = Field(None, description='If a non-blank row has fewer fields than the length of [`fieldnames`][oteapi.strategies.parse.text_csv.ReaderConfig.fieldnames], the missing values are filled-in with the value of [`restval`][oteapi.strategies.parse.text_csv.ReaderConfig.restval].')
class-attribute
instance-attribute
¶
resource
¶
resource_url
¶
Strategy class for resource/url.
ResourceURLConfig
¶
Bases: ResourceConfig
Resource URL strategy config.
Source code in oteapi/strategies/resource/resource_url.py
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
|
downloadUrl: HostlessAnyUrl = Field(..., description=ResourceConfig.model_fields['downloadUrl'].description)
class-attribute
instance-attribute
¶
mediaType: str = Field(..., description=ResourceConfig.model_fields['mediaType'].description)
class-attribute
instance-attribute
¶
resourceType: Literal['resource/url'] = Field('resource/url', description=ResourceConfig.model_fields['resourceType'].description)
class-attribute
instance-attribute
¶
ResourceURLStrategy
¶
Basic resource strategy targeting downloadUrl resources.
Source code in oteapi/strategies/resource/resource_url.py
35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
|
resource_config: ResourceURLConfig
instance-attribute
¶
get()
¶
resource distribution.
Source code in oteapi/strategies/resource/resource_url.py
45 46 47 48 49 50 51 |
|
initialize()
¶
Initialize.
Source code in oteapi/strategies/resource/resource_url.py
41 42 43 |
|
transformation
¶
celery_remote
¶
Transformation Plugin that uses the Celery framework to call remote workers.
CELERY_APP = Celery(broker=f'redis://{REDIS_HOST}:{REDIS_PORT}', backend=f'redis://{REDIS_HOST}:{REDIS_PORT}')
module-attribute
¶
REDIS_HOST = os.getenv('OTEAPI_REDIS_HOST', 'redis')
module-attribute
¶
REDIS_PORT = int(os.getenv('OTEAPI_REDIS_PORT', '6379'))
module-attribute
¶
CeleryConfig
¶
Bases: AttrDict
Celery configuration.
All fields here (including those added from the session through the get()
method,
as well as those added "anonymously") will be used as keyword arguments to the
send_task()
method for the Celery App.
Note
Using alias
for the name
field to favor populating it with task_name
arguments, since this is the "original" field name. I.e., this is done for
backwards compatibility.
Special pydantic configuration settings:
populate_by_name
Allow populating CeleryConfig.name usingname
as well astask_name
.
Source code in oteapi/strategies/transformation/celery_remote.py
35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
|
args: list = Field(..., description='List of arguments for the task.')
class-attribute
instance-attribute
¶
model_config = ConfigDict(populate_by_name=True)
class-attribute
instance-attribute
¶
name: str = Field(..., description='A task name.', alias='task_name')
class-attribute
instance-attribute
¶
CeleryContent
¶
Bases: AttrDict
Class for returning values from a Celery task.
Source code in oteapi/strategies/transformation/celery_remote.py
60 61 62 63 |
|
celery_task_id: str = Field(..., description='A Celery task identifier.')
class-attribute
instance-attribute
¶
CeleryRemoteStrategy
¶
Submit job to remote Celery runner.
Registers strategies:
("transformationType", "celery/remote")
Source code in oteapi/strategies/transformation/celery_remote.py
78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 |
|
transformation_config: CeleryStrategyConfig
instance-attribute
¶
get()
¶
Run a job, return a job ID.
Source code in oteapi/strategies/transformation/celery_remote.py
90 91 92 93 94 95 96 |
|
initialize()
¶
Initialize a job.
Source code in oteapi/strategies/transformation/celery_remote.py
98 99 100 |
|
status(task_id)
¶
Get job status.
Source code in oteapi/strategies/transformation/celery_remote.py
102 103 104 105 |
|
CeleryStrategyConfig
¶
Bases: TransformationConfig
Celery strategy-specific configuration.
Source code in oteapi/strategies/transformation/celery_remote.py
66 67 68 69 70 71 72 73 74 75 |
|