Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • djangoldp-packages/djangoldp
  • decentral1se/djangoldp
  • femmefaytale/djangoldp
  • jvtrudel/djangoldp
4 results
Show changes
Commits on Source (568)
468 additional commits have been omitted to prevent performance issues.
Hide whitespace changes
Inline Side-by-side
Showing
with 1025 additions and 566 deletions
.gitignore
View file @ e97d33ad
dist
build
*.egg-info
*.eggs
*.pyc
*~
*.swp
djangoldp/tests/tests_temp.py
*/.idea/*
.DS_STORE
venv
.gitlab-ci.yml
View file @ e97d33ad
---
image: python:3.6
image: python:3.11
stages:
- test
- release
include:
project: infra/gitlab
ref: master
file: templates/python.ci.yml
test:
stage: test
......@@ -11,22 +12,22 @@ test:
- pip install .[dev]
- python -m unittest djangoldp.tests.runner
except:
- master
- tags
tags:
- test
publish:
stage: release
before_script:
- pip install python-semantic-release~=5.0 sib-commit-parser~=0.3
- git config user.name "${GITLAB_USER_NAME}"
- git config user.email "${GITLAB_USER_EMAIL}"
- git remote set-url origin "https://gitlab-ci-token:${GL_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git"
- git fetch --tags
crypto-test:
stage: test
script:
- semantic-release publish
only:
- master
- pip install .[crypto]
- python -m unittest djangoldp_crypto.tests.runner
except:
- tags
tags:
- deploy
- test
publish:
cache: []
extends: .publish_pypi
CHANGELOG.md 0 → 100644
View file @ e97d33ad
# Changelog
<!--next-version-placeholder-->
## v3.0.5 (2023-07-24)
### Fix
* Readme ([`8792530`](https://github.com/djangoldp-packages/djangoldp/commit/87925305b2093282230511bceb38978db4b279a2))
## v3.0.4 (2023-07-24)
### Fix
* Readme ([`eeedc33`](https://github.com/djangoldp-packages/djangoldp/commit/eeedc3378ed3f4e454377431da6bb50202efdcdc))
## v3.0.3 (2023-07-24)
### Fix
* Readme ([`73597b6`](https://github.com/djangoldp-packages/djangoldp/commit/73597b65430a4d23306f78def0331bda60857493))
## Unreleased
* Imported CLI along with development template from sib-manager project
Dockerfile 0 → 100644
View file @ e97d33ad
# use with your own settings.yml
FROM python:3.11
LABEL maintainer="Plup <plup@plup.io>"
# get server
RUN pip install djangoldp
# create a server instance
RUN djangoldp initserver ldpserver
WORKDIR /ldpserver
#COPY settings.yml .
RUN djangoldp install
RUN djangoldp configure --with-dummy-admin
# run the server
EXPOSE 8000
CMD ["djangoldp", "runserver"]
LICENSE
View file @ e97d33ad
MIT License
Copyright (c) 2018 Startin blox
Copyright (c) 2018-2023 Startin blox
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
......
MANIFEST.in 0 → 100644
View file @ e97d33ad
graft djangoldp/conf/package_template
graft djangoldp/conf/server_template
graft djangoldp/templates/
README.md
View file @ e97d33ad
## Synopsis
This module is an add-on for Django REST Framework that serves a django model respecting the Linked Data Platform convention.
It aims at enabling people with little development skills to serve their own data, to be used with a LDP application.
Building a Startin' Blox application? Read this: https://git.happy-dev.fr/startinblox/devops/doc
## Requirements
* Django (known to work with 2.2)
* Django Rest Framework
* pyld==1.0.5
* django-guardian
* djangorestframework-guardian
# Setup a DjangoLDP server
## Installation
Check the [official documentation](https://docs.startinblox.com/import_documentation/djangoldp_guide/install-djangoldp-server.html).
1. Install this module and all its dependencies
## Settings
```bash
$ pip install djangoldp
```
2. Create a django project
```bash
$ django-admin startproject myldpserver
```
3. Add DjangoLDP to INSTALLED_APPS
```python
INSTALLED_APPS = [
...
# make sure all of your own apps are installed BEFORE DjangoLDP
'djangoldp.apps.DjangoldpConfig',
]
```
IMPORTANT: DjangoLDP will register any models which haven't been registered, with the admin. As such it is important to add your own apps above DjangoLDP, so that you can use custom Admin classes if you wish
### User model requirements
When implementing authentication in your own application, you have two options:
* Using or extending [DjangoLDP-Account](https://git.startinblox.com/djangoldp-packages/djangoldp-account), a DjangoLDP package modelling federated users
* Using your own user model & defining the authentication behaviour yourself
Please see the [Authentication guide](https://git.startinblox.com/djangoldp-packages/djangoldp/wikis/guides/authentication) for full information
If you're going to use your own model then for federated login to work your user model must extend `DjangoLDP.Model`, or define a `urlid` field on the user model, for example:
```python
urlid = LDPUrlField(blank=True, null=True, unique=True)
```
If you don't include this field, then all users will be treated as users local to your instance
The `urlid` field is used to uniquely identify the user and is part of the Linked Data Protocol standard. For local users it can be generated at runtime, but for some resources which are from distant servers this is required to be stored
## Creating your first model
1. Create your django model inside a file myldpserver/myldpserver/models.py
Note that container_path will be use to resolve instance iri and container iri
In the future it could also be used to auto configure django router (e.g. urls.py)
```python
from djangoldp.models import Model
* `OIDC_ACCESS_CONTROL_ALLOW_HEADERS`: overrides the access control headers allowed in the `Access-Control-Allow-Headers` CORS header of responses. Defaults to `authorization, Content-Type, if-match, accept, DPoP`
* `ANONYMOUS_USER_NAME` a setting inherited from dependency [Django-Guardian](https://django-guardian.readthedocs.io/en/stable/overview.html)
* `DJANGOLDP_PERMISSIONS`: overrides the list of all permissions on all resources
* `SERIALIZER_CACHE`: toggles the use of a built-in cache in the serialization of containers/resources
* `MAX_RECORDS_SERIALIZER_CACHE`: sets the maximum number of serializer cache records, at which point the cache will be cleared (reset). Defaults to 10,000
* `SEND_BACKLINKS`: enables the searching and sending of [Activities](https://git.startinblox.com/djangoldp-packages/djangoldp/-/wikis/guides/federation) to distant resources linked by users to this server
* `MAX_ACTIVITY_RESCHEDULES`, `DEFAULT_BACKOFF_FACTOR`, `DEFAULT_ACTIVITY_DELAY`, `DEFAULT_REQUEST_TIMEOUT` tweaks the behaviour of the ActivityQueueService
* `STORE_ACTIVITIES`: sets whether to store activities sent and backlinks received, or to treat them as transient (value should be `"VERBOSE"`, `"ERROR"` or `None`). Defaults to `"ERROR"`
* `MAX_RECORDS_ACTIVITY_CACHE`: sets the maximum number of serializer cache records, at which point the cache will be cleared (reset). If set to 0 disables the cache. Defaults to 10,000
* `ENABLE_SWAGGER_DOCUMENTATION`: enables the automatic OpenAPI-based API schema and documentation generation, made available at `http://yourserver/docs/` is the flag is set to True. Default to False
* `DISABLE_LOCAL_OBJECT_FILTER`: disabled the LocalObjectBackendFilter which is processing-time costly and only need activation in federated architecture, so we preferred to add a way to disable it as a workaround for in-progress performances improvements. Default to False
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
```
1.1. Configure container path (optional)
By default it will be "todos/" with an S for model called Todo
```python
<Model>._meta.container_path = "/my-path/"
```
1.2. Configure field visibility (optional)
Note that at this stage you can limit access to certain fields of models using
```python
<Model>._meta.serializer_fields (<>list of field names to show>)
```
For example, if you have a model with a related field with type **django.contrib.auth.models.User** you don't want to show personal details or password hashes.
E.g.
```python
from django.contrib.auth.models import User
User._meta.serializer_fields = ('username','first_name','last_name')
```
Note that this will be overridden if you explicitly set the fields= parameter as an argument to LDPViewSet.urls(), and filtered if you set the excludes= parameter.
2. Add a url in your urls.py:
```python
from django.conf.urls import url
from django.contrib import admin
from djangoldp.views import LDPViewSet
from .models import Todo
urlpatterns = [
url(r'^', include('djangoldp.urls')),
url(r'^admin/', admin.site.urls), # Optional
]
```
This creates 2 routes for each Model, one for the list, and one with an ID listing the detail of an object.
You could also only use this line in settings.py instead:
```python
ROOT_URLCONF = 'djangoldp.urls'
```
## Synopsis
3. In the settings.py file, add your application name at the beginning of the application list, and add the following lines
This module is an add-on for Django REST Framework that serves a django model respecting the Linked Data Platform convention.
```python
STATIC_ROOT = os.path.join(os.path.dirname(BASE_DIR), 'static')
LDP_RDF_CONTEXT = 'https://cdn.happy-dev.fr/owl/hdcontext.jsonld'
DJANGOLDP_PACKAGES = []
SITE_URL = 'http://localhost:8000'
BASE_URL = SITE_URL
```
It aims at enabling people with little development skills to serve their own data, to be used with a LDP application.
* `LDP_RDF_CONTEXT` tells DjangoLDP where our RDF [ontology](https://www.w3.org/standards/semanticweb/ontology) is defined, which will be returned as part of our views in the 'context' field. This is a web URL and you can visit the value to view the full ontology online. The ontology can be a string, as in the example, but it can also be a dictionary, or a list of ontologies (see the [JSON-LD spec](https://json-ld.org) for examples)
* `DJANGOLDP_PACKAGES` defines which other [DjangoLDP packages](https://git.happy-dev.fr/startinblox/djangoldp-packages) we're using in this installation
* `SITE_URL` is the URL serving the site, e.g. `https://example.com/`. Note that if you include the DjangoLDP urls in a nested path (e.g. `https://example.com/api/`), then `SITE_URL` will need to be set to this value
* `BASE_URL` may be different from SITE_URL, e.g. `https://example.com/app/`
## Check technical documentation
* [Using DjangoLDP with your models](./docs/create_model.md)
4. You can also register your model for the django administration site
## Contribute to DjangoLDP
```python
from django.contrib import admin
from djangoldp.admin import DjangoLDPAdmin
from .models import Todo
### Testing
admin.site.register(Todo, DjangoLDPAdmin)
```
Packaged with DjangoLDP is a tests module, containing unit tests
5. You then need to have your WSGI server pointing on myldpserver/myldpserver/wsgi.py
You can extend these tests and add your own test cases by following the examples in the code. You can then run your tests with:
`python -m unittest djangoldp.tests.runner`
6. You will probably need to create a super user
## Check your datas integrity
```bash
$ ./manage.py createsuperuser
```
Because of the way the DjangoLDP's federation work, you can reach some integrity issue within your datas.
7. If you have no CSS on the admin screens :
You can check them with:
```bash
$ ./manage.py collectstatic
./manage.py check_integrity
```
## Execution
To start the server, `cd` to the root of your Django project and run :
You can ignore some servers:
```bash
$ python3 manage.py runserver
```
## Using DjangoLDP
### Models
To use DjangoLDP in your models you just need to extend djangoldp.Model
The Model class allows you to use your models in federation, adding a `urlid` field, and some key methods useful in federation
If you define a Meta for your Model, you will [need to explicitly inherit Model.Meta](https://docs.djangoproject.com/fr/2.2/topics/db/models/#meta-inheritance) in order to inherit the default settings, e.g. `default_permissions`
```python
from djangoldp.models import Model, LDPMetaMixin
class Todo(Model):
name = models.CharField(max_length=255)
class Meta(Model.Meta):
```
See "Custom Meta options" below to see some helpful ways you can tweak the behaviour of DjangoLDP
Your model will be automatically detected and registered with an LDPViewSet and corresponding URLs, as well as being registered with the Django admin panel. If you register your model with the admin panel manually, make sure to extend djangoldp.DjangoLDPAdmin so that the model is registered with [Django-Guardian object permissions](https://django-guardian.readthedocs.io/en/stable/userguide/admin-integration.html). An alternative version which extends Django's `UserAdmin` is available as djangoldp.DjangoLDPUserAdmin
#### Model Federation
Model `urlid`s can be **local** (matching `settings.SITE_URL`), or **external**
To maintain consistency between federated servers, [Activities](https://www.w3.org/TR/activitystreams-vocabulary) such as Create, Update, Delete are sent to external resources referenced in a ForeignKey relation, instructing them on how to manage the reverse-links with the local server
This behaviour can be disabled in settings.py
```python
SEND_BACKLINKS = False
```
It can also be disabled on a model instance
```python
instance.allow_create_backlinks = False
```
### LDPManager
DjangoLDP Models override `models.Manager`, accessible by `Model.objects`
#### local()
For situations where you don't want to include federated resources in a queryset e.g.
```python
Todo.objects.create(name='Local Todo')
Todo.objects.create(name='Distant Todo', urlid='https://anotherserversomewhere.com/todos/1/')
Todo.objects.all() # query set containing { Local Todo, Distant Todo }
Todo.objects.local() # { Local Todo } only
./manage.py check_integrity --ignore "https://server/,https://another-server/"
```
For Views, we also define a FilterBackend to achieve the same purpose. See the section on ViewSets for this purpose
#### nested_fields()
returns a list of all nested field names for the model, built of a union of the model class' `nested_fields` setting, the to-many relations on the model, excluding all fields detailed by `nested_fields_exclude`
### Add you own commands to the `check_integrity` from your own package
## LDPViewSet
DjangoLDP automatically generates ViewSets for your models, and registers these at urls, according to the settings configured in the model Meta (see below for options)
### Custom Parameters
#### lookup_field
Can be used to use a slug in the url instead of the primary key.
Create a `check_integrity.py` file within your app folder containing:
```python
LDPViewSet.urls(model=User, lookup_field='username')
```
#### nested_fields
list of ForeignKey, ManyToManyField, OneToOneField and their reverse relations. When a field is listed in this parameter, a container will be created inside each single element of the container.
def add_arguments(parser):
parser.add_argument(
"--my-own-argument",
default=False,
nargs="?",
const=True,
help="Some help text",
)
In the following example, besides the urls `/members/` and `/members/<pk>/`, two other will be added to serve a container of the skills of the member: `/members/<pk>/skills/` and `/members/<pk>/skills/<pk>/`
```python
<Model>._meta.nested_fields=["skills"]
def check_integrity(options):
if(options["my_own_argument"]):
print("You ran a check_integrity with --my-own-argument!")
else:
print("Run me with `./manage.py check_integrity --my-own-argument`")
```
## Filter Backends
To achieve federation, DjangoLDP includes links to objects from federated servers and stores these as local objects (see 1.0 - Models). In some situations, you will want to exclude these from the queryset of a custom view
To provide for this need, there is defined in `djangoldp.filters` a FilterBackend which can be included in custom viewsets to restrict the queryset to only objects which were created locally:
```python
from djangoldp.filters import LocalObjectFilterBackend
class MyViewSet(..):
filter_backends=[LocalObjectFilterBackend]
```
By default, LDPViewset applies filter backends from the `permission_classes` defined on the model (see 3.1 for configuration)
By default, `LDPViewSets` use another FilterBackend, `LocalObjectOnContainerPathBackend`, which ensures that only local objects are returned when the path matches that of the Models `container_path` (e.g. /users/ will return a list of local users). In very rare situations where this might be undesirable, it's possible to extend `LDPViewSet` and remove the filter_backend:
```python
class LDPSourceViewSet(LDPViewSet):
model = LDPSource
filter_backends = []
```
Following this you will need to update the model's Meta to use the custom `view_set`:
```python
class Meta:
view_set = LDPSourceViewSet
```
## Custom Meta options on models
### rdf_type
Indicates the type the model corresponds to in the ontology. E.g. where `'hd:circle'` is defined in an ontology from `settings.LDP_RDF_CONTEXT`
```python
rdf_type = 'hd:circle'
```
### rdf_context
Sets added `context` fields to be serialized with model instances
```python
rdf_context = {'picture': 'foaf:depiction'}
```
### auto_author
This property allows to associate a model with the logged in user.
```python
class MyModel(models.Model):
author_user = models.ForeignKey(settings.AUTH_USER_MODEL)
class Meta:
auto_author = 'author_user'
```
Now when an instance of `MyModel` is saved, its `author_user` property will be set to the authenticated user.
### auto_author_field
Set this property to make the value of the `auto_author` field a property on the authenticated use.
```python
class MyModel(models.Model):
author_user = models.ForeignKey(settings.AUTH_USER_MODEL)
class Meta:
auto_author = 'author_user'
auto_author_field = 'profile'
```
Now when an instance of `MyModel` is saved, its `author_user` property will be set to the **profile** of the authenticated user.
## permissions
Django-Guardian is used by default to support object-level permissions. Custom permissions can be added to your model using this attribute. See the [Django-Guardian documentation](https://django-guardian.readthedocs.io/en/stable/userguide/assign.html) for more information
## permissions_classes
This allows you to add permissions for anonymous, logged in user, author ... in the url:
By default `LDPPermissions` is used.
Specific permissin classes can be developed to fit special needs.
## anonymous_perms, user_perms, owner_perms
Those allow you to set permissions from your model's meta.
You can give the following permission to them:
* `view`
* `add`
* `change`
* `control`
* `delete`
* `inherit`
With inherit, Users can herit from Anons. Also Owners can herit from Users.
Eg. with this model Anons can view, Auths can add & Owners can edit & delete.
Note that `owner_perms` need a `owner_field` meta that point the field with owner user.
```python
from djangoldp.models import Model
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
user = models.ForeignKey(settings.AUTH_USER_MODEL)
class Meta:
anonymous_perms = ['view']
authenticated_perms = ['inherit', 'add']
owner_perms = ['inherit', 'change', 'control', 'delete']
owner_field = 'user'
```
Important note:
If you need to give permissions to owner's object, don't forget to add auto_author in model's meta
### view_set
In case of custom viewset, you can use
```python
from djangoldp.models import Model
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
class Meta:
view_set = TodoViewSet
```
### container_path
See 3.1. Configure container path (optional)
### serializer_fields
```python
from djangoldp.models import Model
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
class Meta:
serializer_fields = ['name']
```
Only `name` will be serialized
### serializer_fields_exclude
```python
from djangoldp.models import Model
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
class Meta:
serializer_fields_exclude = ['name']
```
Only `deadline` will be serialized
This is achieved when `LDPViewSet` sets the `exclude` property on the serializer in `build_serializer` method. Note that if you use a custom viewset which does not extend LDPSerializer then you will need to set this property yourself
### nested_fields -- DEPRECIATED
Set on a model to auto-generate viewsets and containers for nested relations (e.g. `/circles/<pk>/members/`)
Depreciated in DjangoLDP 0.8.0, as all to-many fields are included as nested fields by default
### nested_fields_exclude
```python
<Model>._meta.nested_fields_exclude=["skills"]
```
Will exclude the field `skills` from the model's nested fields, and prevent a container `/model/<pk>/skills/` from being generated
## Custom urls
To add customs urls who can not be add through the `Model` class, it's possible de create a file named `djangoldp_urls.py`. It will be executed like an `urls.py` file
## Pagination
To enable pagination feature just add this configuration to the server `settings.py` :
```python
REST_FRAMEWORK = {
'DEFAULT_PAGINATION_CLASS': 'djangoldp.pagination.LDPPagination',
'PAGE_SIZE': 20
}
```
## Sources
To enable sources auto creation for all models, change `djangoldp` by `djangoldp.apps.DjangoldpConfig`, on `INSTALLED_APPS`
```python
INSTALLED_APPS = [
'djangoldp.apps.DjangoldpConfig',
]
```
## 301 on domain mismatch
To enable 301 redirection on domain mismatch, add `djangoldp.middleware.AllowOnlySiteUrl` on `MIDDLEWARE`
This ensure that your clients will use `SITE_URL` and avoid mismatch betwen url & the id of a resource/container
```python
MIDDLEWARE = [
'djangoldp.middleware.AllowOnlySiteUrl',
]
```
Notice tht it'll redirect only HTTP 200 Code.
## Extending DjangoLDP
### Testing
Packaged with DjangoLDP is a tests module, containing unit tests
You can extend these tests and add your own test cases by following the examples in the code. You can then run your tests with:
`python -m unittest tests.runner`
You can see a sample on the `check_integrity.py` file of DjangoLDP.
## License
......
djangoldp/__init__.py
View file @ e97d33ad
from django.db.models import options
__version__ = '0.0.0'
options.DEFAULT_NAMES += (
'lookup_field', 'rdf_type', 'rdf_context', 'auto_author', 'auto_author_field', 'owner_field', 'view_set',
'container_path', 'permission_classes', 'serializer_fields', 'serializer_fields_exclude', 'nested_fields',
'nested_fields_exclude', 'depth', 'anonymous_perms', 'authenticated_perms', 'owner_perms')
default_app_config = 'djangoldp.apps.DjangoldpConfig'
'lookup_field', 'rdf_type', 'rdf_context', 'auto_author', 'owner_field', 'owner_urlid_field',
'view_set', 'container_path', 'permission_classes', 'serializer_fields', 'serializer_fields_exclude', 'empty_containers',
'nested_fields', 'depth', 'permission_roles', 'inherit_permissions', 'public_field', 'static_version', 'static_params', 'active_field', 'disable_url')
djangoldp/activities/services.py
View file @ e97d33ad
import threading
import json
import time
import copy
import requests
from queue import Queue
from requests.exceptions import Timeout, ConnectionError
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
from urllib.parse import urlparse
from django.contrib.auth import get_user_model
from django.db.models.signals import post_save, post_delete, m2m_changed
from django.db.models import Q
from django.dispatch import receiver, Signal
from django.conf import settings
from rest_framework.utils import model_meta
......@@ -31,12 +31,92 @@ SCHEDULER_SETTINGS = {
MAX_ACTIVITY_RESCHEDULES = getattr(settings, 'MAX_ACTIVITY_RESCHEDULES', 3)
DEFAULT_BACKOFF_FACTOR = getattr(settings, 'DEFAULT_BACKOFF_FACTOR', 1)
DEFAULT_ACTIVITY_DELAY = getattr(settings, 'DEFAULT_ACTIVITY_DELAY', 3)
DEFAULT_ACTIVITY_DELAY = getattr(settings, 'DEFAULT_ACTIVITY_DELAY', 0.1)
DEFAULT_REQUEST_TIMEOUT = getattr(settings, 'DEFAULT_REQUEST_TIMEOUT', 10)
MAX_RECORDS_ACTIVITY_CACHE = getattr(settings, 'MAX_RECORDS_ACTIVITY_CACHE', 10000)
activity_sending_finished = Signal()
class ActivityInMemoryCache:
'''
{
'urlid': {
# if an object is sent without a urlid it is not cached
'object_urlid': {
# for create/update, just interested in the most recent activity on this external id
'./': ACTIVITY
# for add/remove, we're interested also in the target (container_id)
'circles': ACTIVITY
'circle-members': ACTIVITY
}
}
}
'''
def __init__(self):
self.cache = {
}
def reset(self):
self.cache = {
}
def has(self, urlid, object_id, target_id=None):
if urlid not in self.cache or object_id not in self.cache[urlid]:
return False
if target_id is None:
target_id = './'
return target_id in self.cache[urlid][object_id]
def get(self, urlid, object_id, target_id=None):
if target_id is None:
target_id = './'
if self.has(urlid, object_id, target_id):
return self.cache[urlid][object_id][target_id]
else:
return None
def set(self, urlid, object_id, target_id=None, value=None):
if MAX_RECORDS_ACTIVITY_CACHE == 0:
return
if len(self.cache.keys()) > MAX_RECORDS_ACTIVITY_CACHE:
self.reset()
if target_id is None:
target_id = './'
if urlid not in self.cache:
self.cache[urlid] = {}
if object_id not in self.cache[urlid]:
self.cache[urlid][object_id] = {}
self.cache[urlid][object_id][target_id] = copy.deepcopy(value)
def invalidate(self, urlid, object_id=None, target_id=None):
# can clear the cache for an entire record or at any level in the cache
if object_id is not None:
if target_id is not None:
self.cache[urlid][object_id].pop(target_id, None)
else:
self.cache[urlid].pop(object_id, None)
else:
self.cache.pop(urlid, None)
# used to minimise the activity traffic to necessary activities,
# preferred over a database solution which is slower
ACTIVITY_CACHE = ActivityInMemoryCache()
ACTIVITY_SAVING_SETTING = getattr(settings, 'STORE_ACTIVITIES', 'ERROR')
class ActivityQueueService:
'''Manages an asynchronous queue for Activity format messages'''
initialized = False
......@@ -66,7 +146,7 @@ class ActivityQueueService:
while True:
# wait for queue item to manifest
item = queue.get()
time.sleep(DEFAULT_ACTIVITY_DELAY)
time.sleep(item[2])
cls._activity_queue_worker(item[0], item[1])
cls.queue.task_done()
......@@ -82,7 +162,7 @@ class ActivityQueueService:
cls.revive_activities()
@classmethod
def do_post(cls, url, activity, auth=None):
def do_post(cls, url, activity, auth=None, timeout=DEFAULT_REQUEST_TIMEOUT):
'''
makes a POST request to url, passing activity
:returns: response from server
......@@ -93,7 +173,14 @@ class ActivityQueueService:
if getattr(settings, 'DISABLE_OUTBOX', False) == 'DEBUG':
return {'data': {}}
return requests.post(url, data=json.dumps(activity), headers=headers, timeout=10)
return requests.post(url, data=json.dumps(activity), headers=headers, timeout=timeout)
@classmethod
def _get_str_urlid(cls, obj):
if obj is None:
return None
return obj if isinstance(obj, str) else obj['@id'] if '@id' in obj else None
@classmethod
def _save_activity_from_response(cls, response, url, scheduled_activity):
......@@ -106,13 +193,27 @@ class ActivityQueueService:
if hasattr(response, 'text'):
response_body = response.text
response_location = getattr(response, "Location", None)
status_code = getattr(response, "status_code", None)
success = str(status_code).startswith("2")
return cls._save_sent_activity(scheduled_activity.to_activitystream(), ActivityModel, success=success,
external_id=url, type=scheduled_activity.type,
response_location=response_location, response_code=str(status_code),
response_body=response_body)
status_code = getattr(response, "status_code", response['status_code'] if 'status_code' in response else None)
success = str(status_code).startswith("2") if status_code is not None else False
# strip some key info from the activity (standardise the datatype)
activity_json = scheduled_activity.to_activitystream() if isinstance(scheduled_activity, ScheduledActivity) else scheduled_activity
activity_type = scheduled_activity.type if hasattr(scheduled_activity, 'type') else scheduled_activity['type']
# set the activity cache
if 'object' in activity_json:
object_id = cls._get_str_urlid(activity_json['object'])
if object_id is not None:
target_origin = cls._get_str_urlid(activity_json.get('target', activity_json.get('origin', None)))
ACTIVITY_CACHE.set(url, object_id, target_origin, activity_json)
# save the activity if instructed to do so
if ACTIVITY_SAVING_SETTING == 'VERBOSE' or (not success and ACTIVITY_SAVING_SETTING == 'ERROR'):
return cls._save_sent_activity(activity_json, ActivityModel, success=success,
external_id=url, type=activity_type,
response_location=response_location, response_code=str(status_code),
response_body=response_body)
@classmethod
def _attempt_failed_reschedule(cls, url, scheduled_activity, backoff_factor):
......@@ -178,7 +279,7 @@ class ActivityQueueService:
if type is None:
return []
type = type.lower()
group_a = ['create', 'update', 'delete']
group_a = ['create', 'update', 'delete', 'creation', 'deletion']
groub_b = ['add', 'remove']
if type in group_a:
......@@ -187,9 +288,6 @@ class ActivityQueueService:
return groub_b
return []
def get_fail_response():
return {'response': None, 'activity': None}
types = get_related_activities(scheduled_activity.type)
if len(types) > 0:
scheduled = ScheduledActivity.objects.filter(external_id=scheduled_activity.external_id,
......@@ -201,15 +299,15 @@ class ActivityQueueService:
if len(scheduled) > 0:
scheduled_activity.delete()
return get_fail_response()
return
if scheduled_activity.type == 'update' and not cls._update_is_new(url, scheduled_activity):
scheduled_activity.delete()
return get_fail_response()
return
if scheduled_activity.type in ['add', 'remove'] and not cls._add_remove_is_new(url, scheduled_activity):
scheduled_activity.delete()
return get_fail_response()
return
cls._send_activity(url, scheduled_activity)
......@@ -243,22 +341,17 @@ class ActivityQueueService:
'''returns False if the two activities are equivalent'''
return ordered(old_activity['object']) == ordered(new_activity['object'])
def get_most_recent_sent_activity(external_id):
'''returns the most recently sent activity which meets the specification'''
activities = ActivityModel.objects.filter(external_id=external_id, is_finished=True,
type__in=['create', 'update']).order_by('-created_at')[:10]
for a in activities.all():
a = a.to_activitystream()
if 'object' in a:
return a
return None
def get_most_recent_sent_activity(external_id, object_id):
return ACTIVITY_CACHE.get(external_id, object_id)
# str objects will have to be checked manually by the receiver
new_activity = scheduled_activity.to_activitystream()
if 'object' not in new_activity or isinstance(new_activity['object'], str):
if 'object' not in new_activity or isinstance(new_activity['object'], str) or \
'@id' not in new_activity['object']:
return True
old_activity = get_most_recent_sent_activity(url)
old_activity = get_most_recent_sent_activity(url, new_activity['object']['@id'])
if old_activity is None:
return True
......@@ -271,42 +364,34 @@ class ActivityQueueService:
def _add_remove_is_new(cls, url, scheduled_activity):
'''auxiliary function validates if the receiver does not know about this Add/Remove activity'''
def get_most_recent_sent_activity(source_obj, source_target_origin):
# get a list of activities with the right type
activities = ActivityModel.objects.filter(external_id=url, is_finished=True,
type__in=['add', 'remove']).order_by('-created_at')[:10]
# we are searching for the most recent Add/Remove activity which shares inbox, object and target/origin
for a in activities.all():
astream = a.to_activitystream()
obj = astream.get('object', None)
target_origin = astream.get('target', astream.get('origin', None))
if obj is None or target_origin is None:
continue
obj_id = cls._get_str_urlid(source_obj)
target_id = cls._get_str_urlid(source_target_origin)
if source_obj == obj and source_target_origin == target_origin:
return a
return None
return ACTIVITY_CACHE.get(url, obj_id, target_id)
new_activity = scheduled_activity.to_activitystream()
new_obj = new_activity.get('object', None)
new_target_origin = new_activity.get('target', new_activity.get('origin', None))
# bounds checking
if new_obj is None or new_target_origin is None:
if new_obj is None or new_target_origin is None or \
(not isinstance(new_obj, str) and '@id' not in new_obj) or \
(not isinstance(new_target_origin, str) and '@id' not in new_target_origin):
return True
# if most recent is the same type of activity as me, it's not new
old_activity = get_most_recent_sent_activity(new_obj, new_target_origin)
if old_activity is not None and old_activity.type == scheduled_activity.type:
if old_activity is not None and old_activity['type'].lower() == scheduled_activity.type.lower():
return False
return True
@classmethod
def _push_to_queue(cls, url, scheduled_activity):
def _push_to_queue(cls, url, scheduled_activity, delay=DEFAULT_ACTIVITY_DELAY):
'''wrapper to check for singleton initialization before pushing'''
if not cls.initialized:
cls.start()
cls.queue.put([url, scheduled_activity])
cls.queue.put([url, scheduled_activity, delay])
@classmethod
def resend_activity(cls, url, scheduled_activity, failed=True):
......@@ -323,7 +408,7 @@ class ActivityQueueService:
cls._push_to_queue(url, scheduled_activity)
@classmethod
def send_activity(cls, url, activity, auth=None):
def send_activity(cls, url, activity, auth=None, delay=DEFAULT_ACTIVITY_DELAY):
'''
saves a ScheduledActivity for the parameterised activity and passes it to the queue
:param url: the recipient url inbox
......@@ -337,18 +422,18 @@ class ActivityQueueService:
# schedule the activity
scheduled = cls._save_sent_activity(activity, ScheduledActivity, external_id=url, type=activity.get('type', None))
cls._push_to_queue(url, scheduled)
cls._push_to_queue(url, scheduled, delay)
@classmethod
def _save_sent_activity(cls, activity, model_represenation=ActivityModel, success=False, external_id=None, type=None,
response_location=None, response_code=None, local_id=None, response_body=None):
'''
Auxiliary function saves a record of parameterised activity
:param model_represenation: the model class which should be used to store the activity. Defaults to djangoldp.Activity, must be a subclass
:param model_represenation: the model class which should be used to store the activity. Defaults to djangol.Activity, must be a subclass
'''
payload = bytes(json.dumps(activity), "utf-8")
payload = json.dumps(activity)
if response_body is not None:
response_body = bytes(json.dumps(response_body), "utf-8")
response_body = json.dumps(response_body)
if local_id is None:
local_id = settings.SITE_URL + "/outbox/"
if type is not None:
......@@ -374,7 +459,7 @@ class ActivityPubService(object):
return
obj = {
"@type": Model.get_model_rdf_type(model),
"@type": getattr(model._meta, "rdf_type", None),
"@id": instance.urlid
}
if obj['@type'] is None:
......@@ -386,10 +471,13 @@ class ActivityPubService(object):
value = getattr(instance, field_name, None)
if value is None:
continue
if not hasattr(value, 'urlid'):
continue
sub_object = {
"@id": value.urlid,
"@type": Model.get_model_rdf_type(type(value))
"@type": getattr(value._meta, "rdf_type", None)
}
if sub_object['@type'] is None:
......@@ -508,7 +596,7 @@ class ActivityPubService(object):
info = model_meta.get_field_info(sender)
# bounds checking
if not hasattr(instance, 'urlid') or Model.get_model_rdf_type(sender) is None:
if not hasattr(instance, 'urlid') or getattr(sender._meta, "rdf_type", None) is None:
return set()
# check each foreign key for a distant resource
......@@ -517,7 +605,7 @@ class ActivityPubService(object):
if not relation_info.to_many:
value = getattr(instance, field_name, None)
if value is not None and Model.is_external(value):
target_type = Model.get_model_rdf_type(type(value))
target_type = getattr(value._meta, "rdf_type", None)
if target_type is None:
continue
......@@ -535,9 +623,9 @@ class ActivityPubService(object):
return inboxes
@classmethod
def get_follower_inboxes(cls, object_urlid):
def get_follower_inboxes(cls, object_urlid, object_container=None):
'''Auxiliary function returns a set of inboxes, from the followers of parameterised object urlid'''
inboxes = set(Follower.objects.filter(object=object_urlid).values_list('inbox', flat=True))
inboxes = set(Follower.objects.filter(Q(object=object_urlid) | Q(object__endswith=object_container)).values_list('inbox', flat=True))
return inboxes
@classmethod
......@@ -577,7 +665,7 @@ def check_save_for_backlinks(sender, instance, created, **kwargs):
and not Model.is_external(instance) \
and getattr(instance, 'username', None) != 'hubl-workaround-493':
external_urlids = ActivityPubService.get_related_externals(sender, instance)
inboxes = ActivityPubService.get_follower_inboxes(instance.urlid)
inboxes = ActivityPubService.get_follower_inboxes(instance.urlid, instance.get_container_path())
targets = set().union(ActivityPubService.get_target_inboxes(external_urlids), inboxes)
if len(targets) > 0:
......@@ -600,13 +688,13 @@ def check_save_for_backlinks(sender, instance, created, **kwargs):
def check_delete_for_backlinks(sender, instance, **kwargs):
if getattr(settings, 'SEND_BACKLINKS', True) and getattr(instance, 'allow_create_backlink', False) \
and getattr(instance, 'username', None) != 'hubl-workaround-493':
targets = ActivityPubService.get_follower_inboxes(instance.urlid)
targets = ActivityPubService.get_follower_inboxes(instance.urlid, instance.get_container_path())
if len(targets) > 0:
for target in targets:
ActivityPubService.send_delete_activity(BACKLINKS_ACTOR, {
"@id": instance.urlid,
"@type": Model.get_model_rdf_type(sender)
"@type": getattr(instance._meta, "rdf_type", None)
}, target)
# remove any Followers on this resource
......@@ -642,8 +730,8 @@ def check_m2m_for_backlinks(sender, instance, action, *args, **kwargs):
# we can only send backlinks on pre_clear because on post_clear the objects are gone
if action != "pre_clear" and pk_set is None:
return
member_rdf_type = Model.get_model_rdf_type(member_model)
container_rdf_type = Model.get_model_rdf_type(type(instance))
member_rdf_type = getattr(member_model._meta, "rdf_type", None)
container_rdf_type = getattr(instance._meta, "rdf_type", None)
if member_rdf_type is None:
return
......@@ -652,7 +740,12 @@ def check_m2m_for_backlinks(sender, instance, action, *args, **kwargs):
# build list of targets (models affected by the change)
if action == "pre_clear":
pk_set = sender.objects.all().values_list(member_model.__name__.lower(), flat=True)
sender_info = model_meta.get_field_info(sender)
for field_name, relation_info in sender_info.relations.items():
if relation_info.related_model == member_model:
pk_set = sender.objects.all().values_list(field_name, flat=True)
if pk_set is None or len(pk_set) == 0:
return
query_set = member_model.objects.filter(pk__in=pk_set)
targets = build_targets(query_set)
......
djangoldp/activities/verbs.py
View file @ e97d33ad
......@@ -33,12 +33,24 @@ class Activity(Object):
delattr(new, "bcc")
return new
def _validate_type_id_defined(self, value):
'''recursively ensures that all nested dict items define @id and @type attributes'''
for item in value.items():
if isinstance(item[1], dict):
item_value = item[1]
if '@type' not in item_value or '@id' not in item_value:
raise errors.ActivityStreamValidationError("all sub-objects passed in activity object must define @id and @type tags")
self._validate_type_id_defined(item_value)
def validate(self):
for attr in self.required_attributes.keys():
if not isinstance(getattr(self, attr, None), self.required_attributes[attr]):
raise errors.ActivityStreamValidationError("required attribute " + attr + " of type "
+ str(self.required_attributes[attr]))
# validate that every dictionary stored in object has @id and @type
self._validate_type_id_defined(self.__getattribute__("object"))
class Add(Activity):
type = "Add"
......
djangoldp/admin.py
View file @ e97d33ad
from csv import DictWriter
from django.contrib import admin
from guardian.admin import GuardedModelAdmin
from django.contrib.auth.admin import UserAdmin
from djangoldp.models import Activity, ScheduledActivity
from django.core.exceptions import FieldDoesNotExist
from django.http import HttpResponse
from guardian.admin import GuardedModelAdmin
from djangoldp.models import Activity, ScheduledActivity, Follower
from djangoldp.activities.services import ActivityQueueService
class DjangoLDPAdmin(GuardedModelAdmin):
......@@ -9,13 +13,38 @@ class DjangoLDPAdmin(GuardedModelAdmin):
An admin model representing a federated object. Inherits from GuardedModelAdmin to provide Django-Guardian
object-level permissions
'''
pass
class DjangoLDPUserAdmin(UserAdmin, GuardedModelAdmin):
actions = ['export_csv']
export_fields = []
def resolve_verbose_name(self, field_path):
field = self
for field_name in field_path.split('__'):
try:
field = field.model._meta.get_field(field_name)
except FieldDoesNotExist:
return None
return field.verbose_name
@admin.action(description="Export CSV")
def export_csv(self, request, queryset):
response = HttpResponse(content_type="text/csv")
response['Content-Disposition'] = f'attachment; filename="{self.model.__name__}.csv"'
# only keep fields that can be resolved, keep only urlid if none
field_list = list(filter(self.resolve_verbose_name, self.export_fields or self.list_display)) or ['urlid']
headers = {field:self.resolve_verbose_name(field) for field in field_list}
writer = DictWriter(response, fieldnames=field_list)
writer.writerow(headers)
writer.writerows(queryset.values(*field_list))
return response
class DjangoLDPUserAdmin(UserAdmin, DjangoLDPAdmin):
'''An extension of UserAdmin providing the functionality of DjangoLDPAdmin'''
list_display = ('urlid', 'email', 'first_name', 'last_name', 'date_joined', 'last_login', 'is_staff')
search_fields = ['urlid', 'email', 'first_name', 'last_name']
ordering = ['urlid']
def get_fieldsets(self, request, obj=None):
fieldsets = super().get_fieldsets(request, obj)
......@@ -35,11 +64,21 @@ class DjangoLDPUserAdmin(UserAdmin, GuardedModelAdmin):
return fieldsets
@admin.action(description='Resend activity')
def resend_activity(modeladmin, request, queryset):
for a in queryset:
ActivityQueueService.send_activity(a.external_id, a.to_activitystream())
resend_activity.short_description = 'Resend activity'
@admin.register(Activity, ScheduledActivity)
class ActivityAdmin(DjangoLDPAdmin):
fields = ['urlid', 'type', 'local_id', 'external_id', 'created_at', 'success', 'payload_view', 'response_code',
'response_location', 'response_body_view']
list_display = ['created_at', 'type', 'local_id', 'external_id', 'success', 'response_code']
readonly_fields = ['created_at', 'payload_view', 'response_location', 'response_code', 'response_body_view']
search_fields = ['urlid', 'type', 'local_id', 'external_id', 'response_code']
actions = [resend_activity]
def payload_view(self, obj):
return str(obj.to_activitystream())
......@@ -48,5 +87,10 @@ class ActivityAdmin(DjangoLDPAdmin):
return str(obj.response_to_json())
admin.site.register(Activity, ActivityAdmin)
admin.site.register(ScheduledActivity, ActivityAdmin)
@admin.register(Follower)
class FollowerAdmin(DjangoLDPAdmin):
fields = ['urlid', 'object', 'inbox', 'follower']
list_display = ['urlid', 'object', 'inbox', 'follower']
search_fields = ['object', 'inbox', 'follower']
djangoldp/apps.py
View file @ e97d33ad
......@@ -7,6 +7,18 @@ class DjangoldpConfig(AppConfig):
def ready(self):
self.auto_register_model_admin()
self.start_activity_queue()
# Patch guardian core to avoid prefetching permissions several times
from guardian.core import ObjectPermissionChecker
ObjectPermissionChecker._prefetch_cache_orig = ObjectPermissionChecker._prefetch_cache
def _prefetch_cache(self):
if hasattr(self.user, "_guardian_perms_cache"):
self._obj_perms_cache = self.user._guardian_perms_cache
return
self._prefetch_cache_orig()
ObjectPermissionChecker._prefetch_cache = _prefetch_cache
def start_activity_queue(self):
from djangoldp.activities.services import ActivityQueueService
......@@ -22,6 +34,7 @@ class DjangoldpConfig(AppConfig):
from django.conf import settings
from django.contrib import admin
from djangoldp.admin import DjangoLDPAdmin
from djangoldp.urls import get_all_non_abstract_subclasses
from djangoldp.models import Model
for package in settings.DJANGOLDP_PACKAGES:
......@@ -36,9 +49,6 @@ class DjangoldpConfig(AppConfig):
except ModuleNotFoundError:
pass
model_classes = {cls.__name__: cls for cls in Model.__subclasses__()}
for class_name in model_classes:
model_class = model_classes[class_name]
if not admin.site.is_registered(model_class):
admin.site.register(model_class, DjangoLDPAdmin)
for model in get_all_non_abstract_subclasses(Model):
if not admin.site.is_registered(model):
admin.site.register(model, DjangoLDPAdmin)
djangoldp/check_integrity.py 0 → 100644
View file @ e97d33ad
'''
DjangoLDP Check Integrity
Usage `./manage.py check_integrity --help`
'''
from django.apps import apps
from django.conf import settings
from djangoldp.models import LDPSource
from urllib.parse import urlparse
import requests
# Helper command for argument type checking
def is_string(target):
if isinstance(target, str):
return target
return False
# Helper command to check status code of a target
def is_alive(target, status_code = 200):
return requests.get(target).status_code == status_code
# Add argument to the `check_integrity` command
def add_arguments(parser):
parser.add_argument(
"--ignore",
action="store",
default=False,
type=is_string,
help="Ignore any server, comma separated",
)
parser.add_argument(
"--ignore-faulted",
default=False,
nargs="?",
const=True,
help="Ignore eventual faulted",
)
parser.add_argument(
"--ignore-404",
default=False,
nargs="?",
const=True,
help="Ignore eventual 404",
)
parser.add_argument(
"--fix-faulted-resources",
default=False,
nargs="?",
const=True,
help="Fix faulted resources",
)
parser.add_argument(
"--fix-404-resources",
default=False,
nargs="?",
const=True,
help="Fix 404 resources",
)
parser.add_argument(
"--fix-offline-servers",
default=False,
nargs="?",
const=True,
help="Remove resources from offline servers",
)
# Define our own checks
def check_integrity(options):
models = apps.get_models()
ignored = set()
if(options["ignore"]):
for target in options["ignore"].split(","):
ignored.add(urlparse(target).netloc)
if(len(ignored) > 0):
print("Ignoring servers:")
for server in ignored:
print("- "+server)
resources = set()
resources_map = dict()
base_urls = set()
for model in models:
for obj in model.objects.all():
if hasattr(obj, "urlid"):
if(obj.urlid):
if(not obj.urlid.startswith(settings.BASE_URL)):
url = urlparse(obj.urlid).netloc
if(url not in ignored):
resources.add(obj.urlid)
resources_map[obj.urlid] = obj
base_urls.add(url)
if(len(base_urls) > 0):
print("Found "+str(len(resources_map))+" distant resources on "+str(len(models))+" models")
print("Servers that I have backlinks to:")
for server in base_urls:
print("- "+server)
else:
print("I don't have any backlink")
source_urls = set()
for source in LDPSource.objects.all():
source_urls.add(urlparse(source.urlid).netloc)
if(len(source_urls) > 0):
print("Servers that I'm allowed to get federated to:")
for server in source_urls:
print("- "+server)
else:
print("I'm not federated")
difference_servers = base_urls.difference(source_urls)
if(len(difference_servers) > 0):
print("Servers that I should not get aware of:")
for server in difference_servers:
print("- "+server)
# Handle faulted resources
if(not options["ignore_faulted"]):
faulted_resources = set()
for server in difference_servers:
for resource in resources:
if(urlparse(resource).netloc in server):
faulted_resources.add(resource)
if(len(faulted_resources) > 0):
print("Resources in fault:")
for resource in faulted_resources:
print("- "+resource)
else:
print("No resource are in fault")
if(options["fix_faulted_resources"]):
for resource in faulted_resources:
try:
resources_map[resource].delete()
except:
pass
print("Fixed faulted resources")
else:
print("Fix them with `./manage.py check_integrity --fix-faulted-resources`")
else:
print("I accept datas for every of those servers")
# Handle 404 resources
if(not options["ignore_404"]):
resources_404 = set()
resources_servers_offline = set()
for resource in resources:
try:
if(is_alive(resource, 404)):
resources_404.add(resource)
except:
resources_servers_offline.add(resource)
if(len(resources_404) > 0):
print("Faulted resources, 404:")
for resource in resources_404:
print("- "+resource)
if(options["fix_404_resources"]):
for resource in resources_404:
try:
resources_map[resource].delete()
except:
pass
print("Fixed 404 resources")
else:
print("Fix them with `./manage.py check_integrity --fix-404-resources`")
if(len(resources_servers_offline) > 0):
print("Faulted resources, servers offline:")
for resource in resources_servers_offline:
print("- "+resource)
if(options["fix_offline_servers"]):
for resource in resources_servers_offline:
try:
resources_map[resource].delete()
except:
pass
print("Fixed resources on offline servers")
else:
print("Fix them with `./manage.py check_integrity --fix-offline-servers`")
else:
print("No 404 in known resources")
\ No newline at end of file
djangoldp/cli.py 0 → 100644
View file @ e97d33ad
import click
import sys
import yaml
import subprocess
from pkg_resources import resource_filename
from pathlib import Path
from django.core import management
from django.core.management.base import CommandError
from . import __version__
# click entrypoint
@click.group()
@click.version_option(__version__)
def main():
"""DjangoLDP CLI"""
@main.command()
@click.argument('name', nargs=1, required=False)
def initserver(name):
"""Start a DjangoLDP server."""
try:
# use directly pwd
directory = Path.cwd()
if name:
# create a directory from project name in pwd
directory = Path.cwd() / name
directory.mkdir(parents=False, exist_ok=False)
# get the template path
template = resource_filename(__name__, 'conf/server_template')
# wrap the default django-admin startproject command
# this call import django settings and configure it
# see: https://docs.djangoproject.com/fr/2.2/topics/settings/#calling-django-setup-is-required-for-standalone-django-usage
# see: https://github.com/django/django/blob/stable/2.2.x/django/core/management/templates.py#L108
# fix: in 2.2 gabarit files options has been renamed: https://github.com/django/django/blob/stable/2.2.x/django/core/management/templates.py#L53
management.call_command('startproject', name, directory, template=template, files=['settings.yml'])
except FileExistsError:
click.echo(f'Error: the folder {directory} already exists')
sys.exit(1)
except CommandError as e:
click.echo(f'Error: {e}')
directory.rmdir()
sys.exit(1)
@main.command()
@click.argument('name', nargs=1)
def startpackage(name):
"""Start a DjangoLDP package."""
try:
# set directory
directory = Path.cwd() / name
# get the template path
template = resource_filename(__name__, 'conf/package_template')
# create dir
directory.mkdir(parents=False, exist_ok=False)
# wrap the default startapp command
management.call_command('startapp', name, directory, template=template)
except FileExistsError:
click.echo(f'Error: the folder {directory} already exists')
sys.exit(1)
except CommandError as e:
click.echo(f'Error: {e}')
sys.exit(1)
@main.command()
def install():
"""Install project dependencies."""
try:
# load dependencies from config file
path = Path.cwd() / 'settings.yml'
with open(path, 'r') as f:
dependencies = yaml.safe_load(f).get('dependencies', [])
# install them by calling pip command
cmd = [sys.executable, "-m", "pip", "install", "--upgrade"]
try:
cmd.extend(dependencies)
subprocess.run(cmd).check_returncode()
click.echo('Installation done!')
except TypeError:
click.echo('No dependency to install')
except FileNotFoundError:
click.echo('Config error: no settings.yml file in this directory')
sys.exit(1)
except subprocess.CalledProcessError as e:
click.echo(f'Installation error: {e}')
sys.exit(1)
@main.command()
@click.option('--with-admin', 'admin', help='Create an administrator user with email.')
@click.option('--email', help='Provide an email for administrator.')
@click.option('--with-dummy-admin', 'dummy_admin', is_flag=True, help='Create a default "admin" user.')
def configure(admin, dummy_admin, email):
"""Configure the project."""
try:
# shortcut to the djangoldp.management command
path = str(Path.cwd() / 'manage.py')
cmd = [sys.executable, path, 'configure']
if admin:
if not email:
click.echo('Error: missing email for admin user')
return
cmd.extend(['--with-admin', admin, '--email', email])
elif dummy_admin:
cmd.append('--with-dummy-admin')
subprocess.run(cmd).check_returncode()
click.echo('Configuration done!')
except subprocess.CalledProcessError as e:
click.echo(f'Configuration error: {e}')
sys.exit(1)
@main.command()
def runserver():
"""Run the Django embeded webserver."""
try:
# shortcut to the djangoldp.management command
path = str(Path.cwd() / 'manage.py')
cmd = [sys.executable, path, 'runserver', '0.0.0.0:8000']
subprocess.run(cmd).check_returncode()
except subprocess.CalledProcessError as e:
click.echo(f'Execution error: {e}')
sys.exit(1)
djangoldp/conf/__init__.py 0 → 100644
View file @ e97d33ad
djangoldp/conf/default_settings.py 0 → 100644
View file @ e97d33ad
"""This module override some of default django configuration from the global settings."""
from django.conf.global_settings import *
####################
# LDP #
####################
LDP_RDF_CONTEXT = 'https://cdn.startinblox.com/owl/context.jsonld'
MAX_ACTIVITY_RESCHEDULES = 3
DEFAULT_BACKOFF_FACTOR = 1
DEFAULT_ACTIVITY_DELAY = 0.2
DEFAULT_REQUEST_TIMEOUT = 10
LDP_INCLUDE_INNER_PERMS = False
####################
# CORE #
####################
# https://en.wikipedia.org/wiki/List_of_tz_zones_by_name (although not all
# systems may support all possibilities). When USE_TZ is True, this is
# interpreted as the default user time zone.
TIME_ZONE = 'UTC'
# If you set this to True, Django will use timezone-aware datetimes.
USE_TZ = True
# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'en-us'
# Database connection info. If left empty, will default to the dummy backend.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': 'db.sqlite3',
}
}
# List of strings representing installed apps.
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'djangoldp',
'guardian'
]
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
# Default server url
SITE_URL = 'http://localhost:8000'
BASE_URL = SITE_URL
# Default URL conf
ROOT_URLCONF = 'djangoldp.urls'
# Absolute filesystem path to the directory that will hold user-uploaded files.
# Example: "/var/www/example.com/media/"
MEDIA_ROOT = ''
# URL that handles the media served from MEDIA_ROOT.
# Examples: "http://example.com/media/", "http://media.example.com/"
MEDIA_URL = '/media/'
# Absolute path to the directory static files should be collected to.
# Example: "/var/www/example.com/static/"
STATIC_ROOT = None
# URL that handles the static files served from STATIC_ROOT.
# Example: "http://example.com/static/", "http://static.example.com/"
STATIC_URL = '/static/'
# ETAGS config
USE_ETAGS = True
# Default X-Frame-Options header value
X_FRAME_OPTIONS = 'DENY'
# The Python dotted path to the WSGI application that Django's internal server
# (runserver) will use. If `None`, the return value of
# 'django.core.wsgi.get_wsgi_application' is used, thus preserving the same
# behavior as previous versions of Django. Otherwise this should point to an
# actual WSGI application object.
WSGI_APPLICATION = 'server.wsgi.application'
##############
# MIDDLEWARE #
##############
# List of middleware to use. Order is important; in the request phase, these
# middleware will be applied in the order given, and in the response
# phase the middleware will be applied in reverse order.
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'djangoldp.middleware.AllowRequestedCORSMiddleware',
'django.middleware.gzip.GZipMiddleware',
'django_brotli.middleware.BrotliMiddleware'
]
##################
# REST FRAMEWORK #
##################
REST_FRAMEWORK = {
'DEFAULT_SCHEMA_CLASS': 'drf_spectacular.openapi.AutoSchema',
'DEFAULT_PAGINATION_CLASS': 'djangoldp.pagination.LDPPagination'
}
###################
# DRF SPECTACULAR #
###################
ENABLE_SWAGGER_DOCUMENTATION = False
SPECTACULAR_SETTINGS = {
'TITLE': 'DjangoLDP Based API Description',
'DESCRIPTION': 'Here you will find the list of all endpoints available on your Djangoldp-based server instance and\
the available methods, needed parameters and requests examples. The list of available endpoints depend on your instance configuration\
especially the list of bloxes and associated django models activated.',
'VERSION': '2.1.37',
'SERVE_INCLUDE_SCHEMA': False
}
############
# SESSIONS #
############
# Same site policy
DCS_SESSION_COOKIE_SAMESITE = 'none'
##################
# AUTHENTICATION #
##################
AUTHENTICATION_BACKENDS = ['django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend']
OIDC_ACCESS_CONTROL_ALLOW_HEADERS = 'Content-Type, if-match, accept, authorization, DPoP, cache-control, pragma, prefer'
# The minimum number of seconds a password reset link is valid for
PASSWORD_RESET_TIMEOUT = 60 * 60 * 24 * 3
DISABLE_LOCAL_OBJECT_FILTER = False
GUARDIAN_AUTO_PREFETCH = True
DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
djangoldp/conf/ldpsettings.py 0 → 100644
View file @ e97d33ad
import os
import sys
import yaml
import logging
from django.core.exceptions import ImproperlyConfigured
from django.conf import settings as django_settings
from pathlib import Path
from collections import OrderedDict
from typing import Iterable
from importlib import import_module
from . import default_settings
logger = logging.getLogger(__name__)
def configure(filename='settings.yml'):
"""Helper function to configure django from LDPSettings."""
yaml_config = None
try:
with open(filename, 'r') as f:
yaml_config = yaml.safe_load(f)
except FileNotFoundError:
logger.info('Starting project without configuration file')
# ref: https://docs.djangoproject.com/fr/2.2/topics/settings/#custom-default-settings
ldpsettings = LDPSettings(yaml_config)
django_settings.configure(ldpsettings)
class LDPSettings(object):
"""Class managing the DjangoLDP configuration."""
def __init__(self, config):
"""Build a Django Setting object from a dict."""
if django_settings.configured:
raise ImproperlyConfigured('Settings have been configured already')
self._config = config
self._explicit_settings = set(default_settings.__dict__.keys()) #default settings are explicit
self._settings = self.build_settings()
def build_settings(self, extend=['INSTALLED_APPS', 'MIDDLEWARE']):
"""
Look for the parameters in multiple places.
Each step overrides the value of the previous key found. Except for "extend" list. Those value must be lists and all values found are added to these lists without managing duplications.
Resolution order of the configuration:
1. Core default settings
2. Packages settings
3. Code from a local settings.py file
4. YAML config file
"""
# helper loop
def update_with(config):
for setting, value in config.items():
self._explicit_settings.add(setting)
if setting in extend:
settings[setting].extend(value)
elif not setting.startswith('_'):
settings.update({setting: value})
# start from default core settings
settings = default_settings.__dict__.copy()
logger.debug(f'Building settings from core defaults')
# INSTALLED_APPS starts empty
settings['INSTALLED_APPS'] = []
# look settings from packages in the order they are given (local overrides installed)
for pkg in self.DJANGOLDP_PACKAGES:
# FIXME: There is something better to do here with the sys.modules path
try:
# override with values from installed package
mod = import_module(f'{pkg}.djangoldp_settings')
update_with(mod.__dict__)
logger.debug(f'Updating settings from installed package {pkg}')
except ModuleNotFoundError:
pass
try:
# override with values from local package
mod = import_module(f'{pkg}.{pkg}.djangoldp_settings')
update_with(mod.__dict__)
logger.debug(f'Updating settings from local package {pkg}')
except ModuleNotFoundError:
pass
# look in settings.py file in directory
try:
mod = import_module('settings')
update_with(mod.__dict__)
logger.debug(f'Updating settings from local settings.py file')
except ModuleNotFoundError:
pass
# look in YAML config file 'server' section
try:
conf = self._config.get('server', {})
update_with(conf)
logger.debug(f'Updating settings with project config')
except KeyError:
pass
# In the end adds the INSTALLED_APPS from the core
settings['INSTALLED_APPS'].extend(getattr(default_settings,'INSTALLED_APPS'))
return settings
@property
def DJANGOLDP_PACKAGES(self):
"""Returns the list of LDP packages configured."""
pkg = self._config.get('ldppackages', [])
return [] if pkg is None else pkg
@property
def INSTALLED_APPS(self):
"""Return the installed apps and the LDP packages."""
# get ldp packages (they are django apps)
apps = self.DJANGOLDP_PACKAGES.copy()
# add the default apps
apps.extend(self._settings['INSTALLED_APPS'])
# As settings come from different origins duplicuation is likeliy to happen
return list(OrderedDict.fromkeys(apps))
def __getattr__(self, param):
"""Return the requested parameter from cached settings."""
if param.startswith('_') or param.islower():
# raise the django exception for inexistent parameter
raise AttributeError(f'"{param}" is not compliant to django settings format')
try:
return self._settings[param]
except KeyError:
# raise the django exception for inexistent parameter
raise AttributeError(f'no "{param}" parameter found in settings')
def is_overridden(self, setting):
return setting in self._explicit_settings
\ No newline at end of file
djangoldp/conf/package_template/README.md 0 → 100644
View file @ e97d33ad
# {{ app_name }}
djangoldp/conf/package_template/app_name/__init__.py-tpl 0 → 100644
View file @ e97d33ad
__version__ = '0.0.0'
djangoldp/conf/package_template/app_name/admin.py-tpl 0 → 100644
View file @ e97d33ad
from django.contrib import admin