Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • djangoldp-packages/djangoldp
  • decentral1se/djangoldp
  • femmefaytale/djangoldp
  • jvtrudel/djangoldp
4 results
Show changes
Commits on Source (790)
690 additional commits have been omitted to prevent performance issues.
Hide whitespace changes
Inline Side-by-side
Showing
with 2015 additions and 253 deletions
.gitignore
View file @ e97d33ad
dist
build
*.egg-info
*.eggs
*.pyc
*~
*.swp
djangoldp/tests/tests_temp.py
*/.idea/*
.DS_STORE
venv
.gitlab-ci.yml
View file @ e97d33ad
---
image: python:3.6
image: python:3.11
stages:
- test
- release
include:
project: infra/gitlab
ref: master
file: templates/python.ci.yml
test:
stage: test
......@@ -11,21 +12,22 @@ test:
- pip install .[dev]
- python -m unittest djangoldp.tests.runner
except:
- master
- tags
tags:
- sib
- test
publish:
stage: release
before_script:
- git config user.name "${GITLAB_USER_NAME}"
- git config user.email "${GITLAB_USER_EMAIL}"
- git remote set-url origin "https://gitlab-ci-token:${GL_TOKEN}@git.happy-dev.fr/${CI_PROJECT_PATH}.git"
- pip install git+https://github.com/plup/python-semantic-release
- pip install sib-commit-parser
crypto-test:
stage: test
script:
- semantic-release publish
only:
- master
- pip install .[crypto]
- python -m unittest djangoldp_crypto.tests.runner
except:
- tags
tags:
- sib
- test
publish:
cache: []
extends: .publish_pypi
CHANGELOG.md 0 → 100644
View file @ e97d33ad
# Changelog
<!--next-version-placeholder-->
## v3.0.5 (2023-07-24)
### Fix
* Readme ([`8792530`](https://github.com/djangoldp-packages/djangoldp/commit/87925305b2093282230511bceb38978db4b279a2))
## v3.0.4 (2023-07-24)
### Fix
* Readme ([`eeedc33`](https://github.com/djangoldp-packages/djangoldp/commit/eeedc3378ed3f4e454377431da6bb50202efdcdc))
## v3.0.3 (2023-07-24)
### Fix
* Readme ([`73597b6`](https://github.com/djangoldp-packages/djangoldp/commit/73597b65430a4d23306f78def0331bda60857493))
## Unreleased
* Imported CLI along with development template from sib-manager project
Dockerfile 0 → 100644
View file @ e97d33ad
# use with your own settings.yml
FROM python:3.11
LABEL maintainer="Plup <plup@plup.io>"
# get server
RUN pip install djangoldp
# create a server instance
RUN djangoldp initserver ldpserver
WORKDIR /ldpserver
#COPY settings.yml .
RUN djangoldp install
RUN djangoldp configure --with-dummy-admin
# run the server
EXPOSE 8000
CMD ["djangoldp", "runserver"]
LICENSE
View file @ e97d33ad
MIT License
Copyright (c) 2018 Startin blox
Copyright (c) 2018-2023 Startin blox
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
......
MANIFEST.in 0 → 100644
View file @ e97d33ad
graft djangoldp/conf/package_template
graft djangoldp/conf/server_template
graft djangoldp/templates/
README.md
View file @ e97d33ad
## Synopsis
This module is an add-on for Django REST Framework that serves a django model respecting the Linked Data Platform convention.
It aims at enabling people with little development skills to serve their own data, to be used with a LDP application.
## Requirements
* Django (known to work with django 1.11)
* Django Rest Framework
* pyld
* django-guardian
* djangorestframework-guardian
## Installation
1. Install this module and all its dependencies
```
pip install djangoldp
```
2. Create a django project
```
django-admin startproject myldpserver
```
3. Create your django model inside a file myldpserver/myldpserver/models.py
Note that container_path will be use to resolve instance iri and container iri
In the future it could also be used to auto configure django router (e.g. urls.py)
```
from djangoldp.models import Model
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
```
# Setup a DjangoLDP server
3.1. Configure container path (optional)
By default it will be "todos/" with an S for model called Todo
```
<Model>._meta.container_path = "/my-path/"
```
Check the [official documentation](https://docs.startinblox.com/import_documentation/djangoldp_guide/install-djangoldp-server.html).
## Settings
3.2. Configure field visibility (optional)
Note that at this stage you can limit access to certain fields of models using
```
<Model>._meta.serializer_fields (<>list of field names to show>)
```
* `OIDC_ACCESS_CONTROL_ALLOW_HEADERS`: overrides the access control headers allowed in the `Access-Control-Allow-Headers` CORS header of responses. Defaults to `authorization, Content-Type, if-match, accept, DPoP`
* `ANONYMOUS_USER_NAME` a setting inherited from dependency [Django-Guardian](https://django-guardian.readthedocs.io/en/stable/overview.html)
* `DJANGOLDP_PERMISSIONS`: overrides the list of all permissions on all resources
* `SERIALIZER_CACHE`: toggles the use of a built-in cache in the serialization of containers/resources
* `MAX_RECORDS_SERIALIZER_CACHE`: sets the maximum number of serializer cache records, at which point the cache will be cleared (reset). Defaults to 10,000
* `SEND_BACKLINKS`: enables the searching and sending of [Activities](https://git.startinblox.com/djangoldp-packages/djangoldp/-/wikis/guides/federation) to distant resources linked by users to this server
* `MAX_ACTIVITY_RESCHEDULES`, `DEFAULT_BACKOFF_FACTOR`, `DEFAULT_ACTIVITY_DELAY`, `DEFAULT_REQUEST_TIMEOUT` tweaks the behaviour of the ActivityQueueService
* `STORE_ACTIVITIES`: sets whether to store activities sent and backlinks received, or to treat them as transient (value should be `"VERBOSE"`, `"ERROR"` or `None`). Defaults to `"ERROR"`
* `MAX_RECORDS_ACTIVITY_CACHE`: sets the maximum number of serializer cache records, at which point the cache will be cleared (reset). If set to 0 disables the cache. Defaults to 10,000
* `ENABLE_SWAGGER_DOCUMENTATION`: enables the automatic OpenAPI-based API schema and documentation generation, made available at `http://yourserver/docs/` is the flag is set to True. Default to False
* `DISABLE_LOCAL_OBJECT_FILTER`: disabled the LocalObjectBackendFilter which is processing-time costly and only need activation in federated architecture, so we preferred to add a way to disable it as a workaround for in-progress performances improvements. Default to False
For example, if you have a model with a related field with type **django.contrib.auth.models.User** you don't want to show personal details or password hashes.
E.g.
```
from django.contrib.auth.models import User
User._meta.serializer_fields = ('username','first_name','last_name')
```
Note that this will be overridden if you explicitly set the fields= parameter as an argument to LDPViewSet.urls(), and filtered if you set the excludes= parameter.
4. Add a url in your urls.py:
```
from django.conf.urls import url
from django.contrib import admin
from djangoldp.views import LDPViewSet
from .models import Todo
urlpatterns = [
url(r'^', include('djangoldp.urls')),
url(r'^admin/', admin.site.urls), # Optional
]
```
This creates 2 routes for each Model, one for the list, and one with an ID listing the detail of an object.
You could also only use this line in settings.py instead:
```
ROOT_URLCONF = 'djangoldp.urls'
```
## Synopsis
5. In the settings.py file, add your application name at the beginning of the application list, and add the following lines
This module is an add-on for Django REST Framework that serves a django model respecting the Linked Data Platform convention.
```
STATIC_ROOT = os.path.join(os.path.dirname(BASE_DIR), 'static')
LDP_RDF_CONTEXT = 'https://cdn.happy-dev.fr/owl/hdcontext.jsonld'
```
It aims at enabling people with little development skills to serve their own data, to be used with a LDP application.
6. You can also register your model for the django administration site
## Check technical documentation
```
from django.contrib import admin
from .models import Todo
* [Using DjangoLDP with your models](./docs/create_model.md)
admin.site.register(Todo)
```
## Contribute to DjangoLDP
7. You then need to have your WSGI server pointing on myldpserver/myldpserver/wsgi.py
### Testing
8. You will probably need to create a super user
```
./manage.py createsuperuser
```
Packaged with DjangoLDP is a tests module, containing unit tests
9. If you have no CSS on the admin screens :
```
./manage.py collectstatic
```
You can extend these tests and add your own test cases by following the examples in the code. You can then run your tests with:
`python -m unittest djangoldp.tests.runner`
## Execution
To start the server, `cd` to the root of your Django project and run :
```
python3 manage.py runserver
```
## Custom Parameters to LDPViewSet
## Check your datas integrity
### lookup_field
Can be used to use a slug in the url instead of the primary key.
```
LDPViewSet.urls(model=User, lookup_field='username')
```
Because of the way the DjangoLDP's federation work, you can reach some integrity issue within your datas.
### nested_fields
list of ForeignKey, ManyToManyField, OneToOneField and their reverse relations. When a field is listed in this parameter, a container will be created inside each single element of the container.
You can check them with:
In the following example, besides the urls `/members/` and `/members/<pk>/`, two other will be added to serve a container of the skills of the member: `/members/<pk>/skills/` and `/members/<pk>/skills/<pk>/`
```bash
./manage.py check_integrity
```
<Model>._meta.nested_fields=["skills"]
```
## Custom Meta options on models
### rdf_type
### auto_author
This property allows to associate a model with the logged in user.
You can ignore some servers:
```python
class MyModel(models.Model):
author_user = models.ForeignKey(settings.AUTH_USER_MODEL)
class Meta:
auto_author = 'author_user'
```bash
./manage.py check_integrity --ignore "https://server/,https://another-server/"
```
Now when an instance of `MyModel` is saved, its `author_user` property will be set to the current user.
## permissions_classes
This allows you to add permissions for anonymous, logged in user, author ... in the url:
By default `LDPPermissions` is used.
Specific permissin classes can be developed to fit special needs.
### Add you own commands to the `check_integrity` from your own package
## anonymous_perms, user_perms, owner_perms
Those allow you to set permissions from your model's meta.
You can give the following permission to them:
* `view`
* `add`
* `change`
* `control`
* `delete`
* `inherit`
With inherit, Users can herit from Anons. Also Owners can herit from Users.
Eg. with this model Anons can view, Auths can add & Owners can edit & delete.
Create a `check_integrity.py` file within your app folder containing:
```python
from djangoldp.models import Model
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
class Meta:
anonymous_perms = ['view']
authenticated_perms = ['inherit', 'add']
owner_perms = ['inherit', 'change', 'control', 'delete']
```
Important note:
If you need to give permissions to owner's object, don't forget to add auto_author in model's meta
### view_set
In case of custom viewset, you can use
```
from djangoldp.models import Model
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
class Meta:
view_set = TodoViewSet
```
### container_path
See 3.1. Configure container path (optional)
### serializer_fields
```
from djangoldp.models import Model
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
class Meta:
serializer_fields = ['name']
def add_arguments(parser):
parser.add_argument(
"--my-own-argument",
default=False,
nargs="?",
const=True,
help="Some help text",
)
def check_integrity(options):
if(options["my_own_argument"]):
print("You ran a check_integrity with --my-own-argument!")
else:
print("Run me with `./manage.py check_integrity --my-own-argument`")
```
Only `name` will be serialized
## Custom urls
To add customs urls who can not be add through the `Model` class, it's possible de create a file named `djangoldp_urls.py`. It will be executed like an `urls.py` file
## Pagination
To enable pagination feature just add this configuration to the server `settings.py` :
You can see a sample on the `check_integrity.py` file of DjangoLDP.
```
REST_FRAMEWORK = {
'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.LimitOffsetPagination',
'PAGE_SIZE': 20
}
```
## License
Licence MIT
djangoldp/__init__.py
View file @ e97d33ad
from django.db.models import options
__version__ = '0.0.0'
options.DEFAULT_NAMES += ('lookup_field', 'rdf_type', 'rdf_context', 'auto_author', 'view_set', 'container_path', 'permission_classes', 'serializer_fields', 'nested_fields', 'depth', 'anonymous_perms', 'authenticated_perms', 'owner_perms')
options.DEFAULT_NAMES += (
'lookup_field', 'rdf_type', 'rdf_context', 'auto_author', 'owner_field', 'owner_urlid_field',
'view_set', 'container_path', 'permission_classes', 'serializer_fields', 'serializer_fields_exclude', 'empty_containers',
'nested_fields', 'depth', 'permission_roles', 'inherit_permissions', 'public_field', 'static_version', 'static_params', 'active_field', 'disable_url')
djangoldp/activities/__init__.py 0 → 100644
View file @ e97d33ad
from .objects import *
from .verbs import *
from .services import *
djangoldp/activities/errors.py 0 → 100644
View file @ e97d33ad
class ActivityStreamDecodeError(Exception):
pass
class ActivityStreamTypeError(Exception):
pass
class ActivityStreamValidationError(Exception):
pass
djangoldp/activities/objects.py 0 → 100644
View file @ e97d33ad
import json
import requests
from djangoldp.activities import errors
class Object(object):
attributes = ["type", "id", "name", "to"]
type = "Object"
@classmethod
def from_json(cls, json):
return Object(**json)
def __init__(self, obj=None, **kwargs):
if obj:
self.__init__(**obj.to_activitystream())
for key in self.attributes:
value = kwargs.get(key)
if value is None:
continue
if isinstance(value, dict) and value.get("type"):
value = as_activitystream(value)
self.__setattr__(key, value)
def __str__(self):
content = json.dumps(self, default=encode_activitystream)
return "<{type}: {content}>".format(type=self.type, content=content)
def to_json(self, context=False):
'''
converts Object to json
context parameter indicates whether @context should be added to the object (for JSON-LD parsing)
'''
values = {}
for attribute in self.attributes:
value = getattr(self, attribute, None)
if value is None:
continue
if isinstance(value, Object):
value = value.to_json()
# if getattr(value, "__iter__", None):
# value = [item.to_json() for item in value]
values[attribute] = value
to = values.get("to")
if isinstance(to, str):
values["to"] = [to]
elif getattr(to, "__iter__", None):
values["to"] = []
for item in to:
if isinstance(item, str):
values["to"].append(item)
if isinstance(item, Object):
values["to"].append(item.id)
if context:
values["@context"] = "https://www.w3.org/ns/activitystreams"
return values
def to_activitystream(self):
return self
class Actor(Object):
attributes = Object.attributes + [
"target",
"publicKey",
"preferredUsername",
"following",
"followers",
"outbox",
"inbox",
]
type = "Actor"
def send(self, activity):
res = requests.post(self.inbox, json=activity.to_json(context=True))
if res.status_code != 200:
raise Exception
class Person(Actor):
type = "Person"
class Service(Actor):
type = "Service"
class Group(Actor):
type = "Group"
class Collection(Object):
attributes = Object.attributes + ["items", "totalItems"]
type = "Collection"
def __init__(self, iterable=None, **kwargs):
self._items = []
Object.__init__(self, **kwargs)
if iterable is None:
return
self.items = iterable
@property
def items(self):
return self._items
@items.setter
def items(self, iterable):
for item in iterable:
if isinstance(item, Object):
self._items.append(item)
elif isinstance(item, str):
self._items.append(item)
elif getattr(item, "to_activitystream", None):
item = as_activitystream(item.to_activitystream())
self._items.append(item)
else:
raise Exception("invalid ActivityStream object: {item}".format(item=item))
def to_json(self, **kwargs):
json = Object.to_json(self, **kwargs)
items = [item.to_json() if isinstance(item, Object) else item
for item in self.items]
json.update({
"items": items
})
return json
class OrderedCollection(Collection):
attributes = Object.attributes + ["orderedItems", "totalItems"]
type = "OrderedCollection"
@property
def totalItems(self):
return len(self.items)
@totalItems.setter
def totalItems(self, value):
pass
@property
def orderedItems(self):
return self.items
@orderedItems.setter
def orderedItems(self, iterable):
self.items = iterable
def to_json(self, **kwargs):
json = Collection.to_json(self, **kwargs)
json["orderedItems"] = json["items"]
del json["items"]
return json
class Link(Object):
type = 'Link'
attributes = ['link']
def to_json(self, **kwargs):
return self.link
#########
# Utils #
#########
ALLOWED_TYPES = {
"Object": Object,
"Actor": Actor,
"Service": Service,
"Person": Person,
"Link": Link,
"Collection": Collection,
"OrderedCollection": OrderedCollection
}
def as_activitystream(obj):
type = obj.get("type")
if type:
if type in ALLOWED_TYPES:
obj = ALLOWED_TYPES[type](**obj)
return obj
raise errors.ActivityStreamDecodeError("Invalid Type {0}".format(type))
return obj
def encode_activitystream(obj):
if isinstance(obj, Object):
return obj.to_json()
raise errors.ActivityStreamTypeError("Unknown ActivityStream Type")
djangoldp/activities/services.py 0 → 100644
View file @ e97d33ad
import threading
import json
import time
import copy
import requests
from queue import Queue
from requests.exceptions import Timeout, ConnectionError
from urllib.parse import urlparse
from django.contrib.auth import get_user_model
from django.db.models.signals import post_save, post_delete, m2m_changed
from django.db.models import Q
from django.dispatch import receiver, Signal
from django.conf import settings
from rest_framework.utils import model_meta
from djangoldp.models import Model, Follower, ScheduledActivity
from djangoldp.models import Activity as ActivityModel
import logging
logger = logging.getLogger('djangoldp')
BACKLINKS_ACTOR = {
"type": "Service",
"name": "Backlinks Service"
}
SCHEDULER_SETTINGS = {
'apscheduler.timezone': getattr(settings, 'TIME_ZONE', 'UTC'),
}
MAX_ACTIVITY_RESCHEDULES = getattr(settings, 'MAX_ACTIVITY_RESCHEDULES', 3)
DEFAULT_BACKOFF_FACTOR = getattr(settings, 'DEFAULT_BACKOFF_FACTOR', 1)
DEFAULT_ACTIVITY_DELAY = getattr(settings, 'DEFAULT_ACTIVITY_DELAY', 0.1)
DEFAULT_REQUEST_TIMEOUT = getattr(settings, 'DEFAULT_REQUEST_TIMEOUT', 10)
MAX_RECORDS_ACTIVITY_CACHE = getattr(settings, 'MAX_RECORDS_ACTIVITY_CACHE', 10000)
activity_sending_finished = Signal()
class ActivityInMemoryCache:
'''
{
'urlid': {
# if an object is sent without a urlid it is not cached
'object_urlid': {
# for create/update, just interested in the most recent activity on this external id
'./': ACTIVITY
# for add/remove, we're interested also in the target (container_id)
'circles': ACTIVITY
'circle-members': ACTIVITY
}
}
}
'''
def __init__(self):
self.cache = {
}
def reset(self):
self.cache = {
}
def has(self, urlid, object_id, target_id=None):
if urlid not in self.cache or object_id not in self.cache[urlid]:
return False
if target_id is None:
target_id = './'
return target_id in self.cache[urlid][object_id]
def get(self, urlid, object_id, target_id=None):
if target_id is None:
target_id = './'
if self.has(urlid, object_id, target_id):
return self.cache[urlid][object_id][target_id]
else:
return None
def set(self, urlid, object_id, target_id=None, value=None):
if MAX_RECORDS_ACTIVITY_CACHE == 0:
return
if len(self.cache.keys()) > MAX_RECORDS_ACTIVITY_CACHE:
self.reset()
if target_id is None:
target_id = './'
if urlid not in self.cache:
self.cache[urlid] = {}
if object_id not in self.cache[urlid]:
self.cache[urlid][object_id] = {}
self.cache[urlid][object_id][target_id] = copy.deepcopy(value)
def invalidate(self, urlid, object_id=None, target_id=None):
# can clear the cache for an entire record or at any level in the cache
if object_id is not None:
if target_id is not None:
self.cache[urlid][object_id].pop(target_id, None)
else:
self.cache[urlid].pop(object_id, None)
else:
self.cache.pop(urlid, None)
# used to minimise the activity traffic to necessary activities,
# preferred over a database solution which is slower
ACTIVITY_CACHE = ActivityInMemoryCache()
ACTIVITY_SAVING_SETTING = getattr(settings, 'STORE_ACTIVITIES', 'ERROR')
class ActivityQueueService:
'''Manages an asynchronous queue for Activity format messages'''
initialized = False
queue = None
@classmethod
def revive_activities(cls):
'''re-schedules all ScheduledActivities to the queue'''
with cls.queue.mutex:
cls.queue.queue.clear()
scheduled = ScheduledActivity.objects.all()
for activity in scheduled:
if activity.external_id is not None:
cls.resend_activity(str(activity.external_id), activity, failed=False)
else:
activity.delete()
@classmethod
def start(cls):
'''
method checks if there are scheduled activities on the queue and starts them up
Important: this method should only be called in start-up, when you know there are not queue tasks running
otherwise duplicate activities may be sent
'''
def queue_worker(queue):
while True:
# wait for queue item to manifest
item = queue.get()
time.sleep(item[2])
cls._activity_queue_worker(item[0], item[1])
cls.queue.task_done()
if not cls.initialized:
cls.initialized = True
# initialise the queue worker - infinite maxsize
cls.queue = Queue(maxsize=0)
t = threading.Thread(target=queue_worker, args=[cls.queue])
t.setDaemon(True)
t.start()
cls.revive_activities()
@classmethod
def do_post(cls, url, activity, auth=None, timeout=DEFAULT_REQUEST_TIMEOUT):
'''
makes a POST request to url, passing activity
:returns: response from server
:raises: Timeout or ConnectionError if the post could not be made
'''
headers = {'Content-Type': 'application/ld+json'}
logger.debug('[Sender] sending Activity... ' + str(activity))
if getattr(settings, 'DISABLE_OUTBOX', False) == 'DEBUG':
return {'data': {}}
return requests.post(url, data=json.dumps(activity), headers=headers, timeout=timeout)
@classmethod
def _get_str_urlid(cls, obj):
if obj is None:
return None
return obj if isinstance(obj, str) else obj['@id'] if '@id' in obj else None
@classmethod
def _save_activity_from_response(cls, response, url, scheduled_activity):
'''
wrapper to save a finished Activity based on the parameterised response
:return: saved Activity object
'''
response_body = None
if hasattr(response, 'text'):
response_body = response.text
response_location = getattr(response, "Location", None)
status_code = getattr(response, "status_code", response['status_code'] if 'status_code' in response else None)
success = str(status_code).startswith("2") if status_code is not None else False
# strip some key info from the activity (standardise the datatype)
activity_json = scheduled_activity.to_activitystream() if isinstance(scheduled_activity, ScheduledActivity) else scheduled_activity
activity_type = scheduled_activity.type if hasattr(scheduled_activity, 'type') else scheduled_activity['type']
# set the activity cache
if 'object' in activity_json:
object_id = cls._get_str_urlid(activity_json['object'])
if object_id is not None:
target_origin = cls._get_str_urlid(activity_json.get('target', activity_json.get('origin', None)))
ACTIVITY_CACHE.set(url, object_id, target_origin, activity_json)
# save the activity if instructed to do so
if ACTIVITY_SAVING_SETTING == 'VERBOSE' or (not success and ACTIVITY_SAVING_SETTING == 'ERROR'):
return cls._save_sent_activity(activity_json, ActivityModel, success=success,
external_id=url, type=activity_type,
response_location=response_location, response_code=str(status_code),
response_body=response_body)
@classmethod
def _attempt_failed_reschedule(cls, url, scheduled_activity, backoff_factor):
'''
either re-schedules a failed activity or saves its failure state, depending on the number of fails and the
fail policy (MAX_ACTIVITY_RESCHEDULES)
:return: True if it was able to reschedule
'''
if scheduled_activity.failed_attempts < MAX_ACTIVITY_RESCHEDULES:
backoff = backoff_factor * (2 ** (scheduled_activity.failed_attempts - 1))
cls.resend_activity(url, scheduled_activity, backoff)
return True
# no retries left, save the failure state
logger.error('Failed to deliver backlink to ' + str(url) + ' after retrying ' +
str(MAX_ACTIVITY_RESCHEDULES) + ' times')
cls._save_sent_activity(scheduled_activity.to_activitystream(), ActivityModel, success=False, external_id=url,
type=scheduled_activity.type, response_code='408')
return False
@classmethod
def _dispatch_activity_sending_finished(cls, response, saved_activity):
'''sends a 'activity_sending_finished' signal to receivers'''
activity_sending_finished.send(sender=cls, response=response, saved_activity=saved_activity)
@classmethod
def _send_activity(cls, url, scheduled_activity, auth=None, backoff_factor=DEFAULT_BACKOFF_FACTOR):
'''
makes a POST request to url, passing ScheduledActivity instance. reschedules if needed
:param backoff_factor: a factor to use in the extension of waiting for retries. Used both in the RetryStrategy
of requests.post and in rescheduling an activity which timed out
'''
response = None
activity = scheduled_activity.to_activitystream()
try:
response = cls.do_post(url, activity, auth)
except (Timeout, ConnectionError):
if cls._attempt_failed_reschedule(url, scheduled_activity, backoff_factor):
# successfully rescheduled, so skip cleanup for now
return
except Exception as e:
logger.error('Failed to deliver backlink to ' + str(url) + ', was attempting ' + str(activity) +
str(e.__class__) + ': ' + str(e))
saved = None
if response is not None:
saved = cls._save_activity_from_response(response, url, scheduled_activity)
scheduled_activity.delete()
# emit activity finished event
cls._dispatch_activity_sending_finished(response, saved)
@classmethod
def _activity_queue_worker(cls, url, scheduled_activity):
'''
Worker for sending a scheduled activity on the queue. Decides whether to send the activity and then passes to
_send_activity if it is worth it
'''
def get_related_activities(type):
'''returns a list of activity types which should be considered a "match" with the parameterised type'''
if type is None:
return []
type = type.lower()
group_a = ['create', 'update', 'delete', 'creation', 'deletion']
groub_b = ['add', 'remove']
if type in group_a:
return group_a
if type in groub_b:
return groub_b
return []
types = get_related_activities(scheduled_activity.type)
if len(types) > 0:
scheduled = ScheduledActivity.objects.filter(external_id=scheduled_activity.external_id,
created_at__gt=scheduled_activity.created_at,
type__in=types)
# filter to scheduled activities on the same object
scheduled = [s for s in scheduled if cls._is_same_object_target(s, scheduled_activity)]
if len(scheduled) > 0:
scheduled_activity.delete()
return
if scheduled_activity.type == 'update' and not cls._update_is_new(url, scheduled_activity):
scheduled_activity.delete()
return
if scheduled_activity.type in ['add', 'remove'] and not cls._add_remove_is_new(url, scheduled_activity):
scheduled_activity.delete()
return
cls._send_activity(url, scheduled_activity)
@classmethod
def _is_same_object_target(cls, activity_a, activity_b):
def get_property(object, property, default=None):
ret = object.get(property, default)
if isinstance(ret, dict):
ret = ret.get('@id', ret.get('name', None))
return ret
a = activity_a.to_activitystream()
b = activity_b.to_activitystream()
if get_property(a, 'object') != get_property(b, 'object'):
return False
return get_property(a, 'target', a.get('origin', None)) == get_property(b, 'target', b.get('origin', None))
@classmethod
def _update_is_new(cls, url, scheduled_activity):
'''auxiliary function which validates if a scheduled update holds new information, compared to a past success'''
def ordered(obj):
'''recursively sorts nested dictionary objects to standardise ordering in comparison'''
if isinstance(obj, dict):
return sorted((k, ordered(v)) for k, v in obj.items())
else:
return obj
def no_new_changes(old_activity, new_activity):
'''returns False if the two activities are equivalent'''
return ordered(old_activity['object']) == ordered(new_activity['object'])
def get_most_recent_sent_activity(external_id, object_id):
return ACTIVITY_CACHE.get(external_id, object_id)
# str objects will have to be checked manually by the receiver
new_activity = scheduled_activity.to_activitystream()
if 'object' not in new_activity or isinstance(new_activity['object'], str) or \
'@id' not in new_activity['object']:
return True
old_activity = get_most_recent_sent_activity(url, new_activity['object']['@id'])
if old_activity is None:
return True
if no_new_changes(old_activity, new_activity):
return False
return True
@classmethod
def _add_remove_is_new(cls, url, scheduled_activity):
'''auxiliary function validates if the receiver does not know about this Add/Remove activity'''
def get_most_recent_sent_activity(source_obj, source_target_origin):
obj_id = cls._get_str_urlid(source_obj)
target_id = cls._get_str_urlid(source_target_origin)
return ACTIVITY_CACHE.get(url, obj_id, target_id)
new_activity = scheduled_activity.to_activitystream()
new_obj = new_activity.get('object', None)
new_target_origin = new_activity.get('target', new_activity.get('origin', None))
# bounds checking
if new_obj is None or new_target_origin is None or \
(not isinstance(new_obj, str) and '@id' not in new_obj) or \
(not isinstance(new_target_origin, str) and '@id' not in new_target_origin):
return True
# if most recent is the same type of activity as me, it's not new
old_activity = get_most_recent_sent_activity(new_obj, new_target_origin)
if old_activity is not None and old_activity['type'].lower() == scheduled_activity.type.lower():
return False
return True
@classmethod
def _push_to_queue(cls, url, scheduled_activity, delay=DEFAULT_ACTIVITY_DELAY):
'''wrapper to check for singleton initialization before pushing'''
if not cls.initialized:
cls.start()
cls.queue.put([url, scheduled_activity, delay])
@classmethod
def resend_activity(cls, url, scheduled_activity, failed=True):
'''
a variation of send_activity for ScheduledActivity objects
:param url: the recipient url inbox
:param scheduled_activity: a ScheduledActivity object for sending
:param failed: set to True to increment scheduled_activity.failed_attempts, to keep track of the number of resends
'''
if failed:
scheduled_activity.failed_attempts = scheduled_activity.failed_attempts + 1
scheduled_activity.save()
cls._push_to_queue(url, scheduled_activity)
@classmethod
def send_activity(cls, url, activity, auth=None, delay=DEFAULT_ACTIVITY_DELAY):
'''
saves a ScheduledActivity for the parameterised activity and passes it to the queue
:param url: the recipient url inbox
:param activity: an Activity to send
'''
if getattr(settings, 'DISABLE_OUTBOX', False) is not False:
if getattr(settings, 'DISABLE_OUTBOX') == 'DEBUG':
cls._save_sent_activity(activity, ActivityModel, external_id=url, success=True, type=activity.get('type', None),
response_code='201')
return
# schedule the activity
scheduled = cls._save_sent_activity(activity, ScheduledActivity, external_id=url, type=activity.get('type', None))
cls._push_to_queue(url, scheduled, delay)
@classmethod
def _save_sent_activity(cls, activity, model_represenation=ActivityModel, success=False, external_id=None, type=None,
response_location=None, response_code=None, local_id=None, response_body=None):
'''
Auxiliary function saves a record of parameterised activity
:param model_represenation: the model class which should be used to store the activity. Defaults to djangol.Activity, must be a subclass
'''
payload = json.dumps(activity)
if response_body is not None:
response_body = json.dumps(response_body)
if local_id is None:
local_id = settings.SITE_URL + "/outbox/"
if type is not None:
type = type.lower()
elif 'type' in activity and isinstance(activity.get('type'), str):
type = activity.get('type').lower()
obj = model_represenation.objects.create(local_id=local_id, payload=payload, success=success,
external_id=external_id, type=type, response_location=response_location,
response_code=response_code, response_body=response_body)
return obj
class ActivityPubService(object):
'''A service aiding the construction and sending of ActivityStreams notifications'''
@classmethod
def build_object_tree(cls, instance):
'''builds a depth 1 object tree from a parameterised instance, with each branch being an object's urlid and RDF type'''
model = type(instance)
info = model_meta.get_field_info(model)
if not hasattr(instance, 'urlid'):
return
obj = {
"@type": getattr(model._meta, "rdf_type", None),
"@id": instance.urlid
}
if obj['@type'] is None:
return
# append relations
for field_name, relation_info in info.relations.items():
if not relation_info.to_many:
value = getattr(instance, field_name, None)
if value is None:
continue
if not hasattr(value, 'urlid'):
continue
sub_object = {
"@id": value.urlid,
"@type": getattr(value._meta, "rdf_type", None)
}
if sub_object['@type'] is None:
continue
obj[field_name] = sub_object
return obj
@classmethod
def get_actor_from_user_instance(cls, user):
'''Auxiliary function returns valid Actor object from parameterised user instance, None if parameter invalid'''
if isinstance(user, get_user_model()) and hasattr(user, 'urlid'):
return {
'@type': 'foaf:user',
'@id': user.urlid
}
return None
@classmethod
def discover_inbox(cls, target_id):
'''a method which discovers the inbox of the target resource'''
url = urlparse(target_id)
return target_id.replace(url.path, "/") + "inbox/"
@classmethod
def build_activity(self, actor, obj, activity_type='Activity', **kwargs):
'''Auxiliary function returns an activity object with kwargs in the body'''
res = {
"@context": [
"https://www.w3.org/ns/activitystreams",
settings.LDP_RDF_CONTEXT
],
"type": activity_type,
"actor": actor,
"object": obj
}
for kwarg in kwargs:
res.update({kwarg: kwargs[kwarg]})
return res
@classmethod
def send_add_activity(cls, actor, obj, target):
'''
Sends an Add activity
:param actor: a valid Actor object
:param obj: a valid ActivityStreams Object
:param target: an object representing the target collection
'''
summary = str(obj['@id']) + " was added to " + str(target['@id'])
activity = cls.build_activity(actor, obj, activity_type='Add', summary=summary, target=target)
# send request
inbox = ActivityPubService.discover_inbox(target['@id'])
ActivityQueueService.send_activity(inbox, activity)
@classmethod
def send_remove_activity(cls, actor, obj, origin):
'''
Sends a Remove activity
:param actor: a valid Actor object, or a user instance
:param obj: a valid ActivityStreams Object
:param origin: the context the object has been removed from
'''
summary = str(obj['@id']) + " was removed from " + str(origin['@id'])
activity = cls.build_activity(actor, obj, activity_type='Remove', summary=summary, origin=origin)
# send request
inbox = ActivityPubService.discover_inbox(origin['@id'])
ActivityQueueService.send_activity(inbox, activity)
@classmethod
def send_create_activity(cls, actor, obj, inbox):
'''
Sends a Create activity
:param actor: a valid Actor object, or a user instance
:param obj: a valid ActivityStreams Object
:param inbox: the inbox to send the activity to
'''
summary = str(obj['@id']) + " was created"
activity = cls.build_activity(actor, obj, activity_type='Create', summary=summary)
ActivityQueueService.send_activity(inbox, activity)
@classmethod
def send_update_activity(cls, actor, obj, inbox):
'''
Sends an Update activity
:param actor: a valid Actor object, or a user instance
:param obj: a valid ActivityStreams Object
:param inbox: the inbox to send the activity to
'''
summary = str(obj['@id']) + " was updated"
activity = cls.build_activity(actor, obj, activity_type='Update', summary=summary)
ActivityQueueService.send_activity(inbox, activity)
@classmethod
def send_delete_activity(cls, actor, obj, inbox):
'''
Sends a Remove activity
:param actor: a valid Actor object, or a user instance
:param obj: a valid ActivityStreams Object
:param inbox: the inbox to send the activity to
'''
summary = str(obj['@id']) + " was deleted"
activity = cls.build_activity(actor, obj, activity_type='Delete', summary=summary)
ActivityQueueService.send_activity(inbox, activity)
@classmethod
def get_related_externals(cls, sender, instance):
'''Auxiliary function returns a set of urlids of distant resources connected to paramertised instance'''
info = model_meta.get_field_info(sender)
# bounds checking
if not hasattr(instance, 'urlid') or getattr(sender._meta, "rdf_type", None) is None:
return set()
# check each foreign key for a distant resource
targets = set()
for field_name, relation_info in info.relations.items():
if not relation_info.to_many:
value = getattr(instance, field_name, None)
if value is not None and Model.is_external(value):
target_type = getattr(value._meta, "rdf_type", None)
if target_type is None:
continue
targets.add(value.urlid)
return targets
@classmethod
def get_target_inboxes(cls, urlids):
'''Auxiliary function returns a set of inboxes, from a set of target object urlids'''
inboxes = set()
for urlid in urlids:
inboxes.add(ActivityPubService.discover_inbox(urlid))
return inboxes
@classmethod
def get_follower_inboxes(cls, object_urlid, object_container=None):
'''Auxiliary function returns a set of inboxes, from the followers of parameterised object urlid'''
inboxes = set(Follower.objects.filter(Q(object=object_urlid) | Q(object__endswith=object_container)).values_list('inbox', flat=True))
return inboxes
@classmethod
def save_follower_for_target(cls, external_urlid, obj_id):
inbox = ActivityPubService.discover_inbox(external_urlid)
if not Follower.objects.filter(object=obj_id, follower=external_urlid).exists():
Follower.objects.create(object=obj_id, inbox=inbox, follower=external_urlid,
is_backlink=True)
@classmethod
def save_followers_for_targets(cls, external_urlids, obj_id):
'''
saves Follower objects for any external urlid which isn't already following the object in question
:param external_urlids: list of external urlids to populate the follower inbox
:param obj_id: object id to be followed
'''
existing_followers = Follower.objects.filter(object=obj_id).values_list('follower', flat=True)
for urlid in external_urlids:
if urlid not in existing_followers:
Follower.objects.create(object=obj_id, inbox=ActivityPubService.discover_inbox(urlid),
follower=urlid, is_backlink=True)
@classmethod
def remove_followers_for_resource(cls, external_urlid, obj_id):
'''removes all followers which match the follower urlid, obj urlid combination'''
inbox = ActivityPubService.discover_inbox(external_urlid)
for follower in Follower.objects.filter(object=obj_id, follower=external_urlid,
inbox=inbox, is_backlink=True):
follower.delete()
@receiver([post_save])
def check_save_for_backlinks(sender, instance, created, **kwargs):
if getattr(settings, 'SEND_BACKLINKS', True) and getattr(instance, 'allow_create_backlink', False) \
and not Model.is_external(instance) \
and getattr(instance, 'username', None) != 'hubl-workaround-493':
external_urlids = ActivityPubService.get_related_externals(sender, instance)
inboxes = ActivityPubService.get_follower_inboxes(instance.urlid, instance.get_container_path())
targets = set().union(ActivityPubService.get_target_inboxes(external_urlids), inboxes)
if len(targets) > 0:
obj = ActivityPubService.build_object_tree(instance)
actor = BACKLINKS_ACTOR
# Create Activity
if created:
for target in targets:
ActivityPubService.send_create_activity(actor, obj, target)
# Update Activity
else:
for target in targets:
ActivityPubService.send_update_activity(actor, obj, target)
# create Followers to update external resources of changes in future
ActivityPubService.save_followers_for_targets(external_urlids, obj['@id'])
@receiver([post_delete])
def check_delete_for_backlinks(sender, instance, **kwargs):
if getattr(settings, 'SEND_BACKLINKS', True) and getattr(instance, 'allow_create_backlink', False) \
and getattr(instance, 'username', None) != 'hubl-workaround-493':
targets = ActivityPubService.get_follower_inboxes(instance.urlid, instance.get_container_path())
if len(targets) > 0:
for target in targets:
ActivityPubService.send_delete_activity(BACKLINKS_ACTOR, {
"@id": instance.urlid,
"@type": getattr(instance._meta, "rdf_type", None)
}, target)
# remove any Followers on this resource
urlid = getattr(instance, 'urlid', None)
if urlid is not None:
for follower in Follower.objects.filter(object=urlid):
follower.delete()
@receiver([m2m_changed])
def check_m2m_for_backlinks(sender, instance, action, *args, **kwargs):
def build_targets(query_set):
'''analyses parameterised queryset (removed or added members) for backlinks'''
targets = []
for obj in query_set:
condition = Model.is_external(obj) and getattr(obj, 'allow_create_backlink', False)
if action == "post_add":
condition = condition and not getattr(obj, 'is_backlink', True)
if condition:
targets.append({
"@type": member_rdf_type,
"@id": obj.urlid
})
return targets
if getattr(settings, 'SEND_BACKLINKS', True):
member_model = kwargs['model']
pk_set = kwargs['pk_set']
# we can only send backlinks on pre_clear because on post_clear the objects are gone
if action != "pre_clear" and pk_set is None:
return
member_rdf_type = getattr(member_model._meta, "rdf_type", None)
container_rdf_type = getattr(instance._meta, "rdf_type", None)
if member_rdf_type is None:
return
if container_rdf_type is None:
return
# build list of targets (models affected by the change)
if action == "pre_clear":
sender_info = model_meta.get_field_info(sender)
for field_name, relation_info in sender_info.relations.items():
if relation_info.related_model == member_model:
pk_set = sender.objects.all().values_list(field_name, flat=True)
if pk_set is None or len(pk_set) == 0:
return
query_set = member_model.objects.filter(pk__in=pk_set)
targets = build_targets(query_set)
if len(targets) > 0:
obj = {
"@type": container_rdf_type,
"@id": instance.urlid
}
if action == 'post_add':
for target in targets:
ActivityPubService.send_add_activity(BACKLINKS_ACTOR, obj, target)
ActivityPubService.save_follower_for_target(target['@id'], obj['@id'])
elif action == "post_remove" or action == "pre_clear":
for target in targets:
ActivityPubService.send_remove_activity(BACKLINKS_ACTOR, obj, target)
ActivityPubService.remove_followers_for_resource(target['@id'], obj['@id'])
djangoldp/activities/verbs.py 0 → 100644
View file @ e97d33ad
from copy import copy
from djangoldp.activities import errors
from djangoldp.activities.objects import ALLOWED_TYPES, Object, Actor
class Activity(Object):
attributes = Object.attributes + ["actor", "object"]
type = "Activity"
# dictionary defining required attributes -> tuple of acceptable types
required_attributes = {
"actor": (Actor, str),
"object": dict
}
def get_audience(self):
audience = []
for attr in ["to", "bto", "cc", "bcc", "audience"]:
value = getattr(self, attr, None)
if not value:
continue
if isinstance(value, str):
value = [value]
audience += value
return set(audience)
def strip_audience(self):
new = copy(self)
if getattr(new, "bto", None):
delattr(new, "bto")
if getattr(new, "bcc", None):
delattr(new, "bcc")
return new
def _validate_type_id_defined(self, value):
'''recursively ensures that all nested dict items define @id and @type attributes'''
for item in value.items():
if isinstance(item[1], dict):
item_value = item[1]
if '@type' not in item_value or '@id' not in item_value:
raise errors.ActivityStreamValidationError("all sub-objects passed in activity object must define @id and @type tags")
self._validate_type_id_defined(item_value)
def validate(self):
for attr in self.required_attributes.keys():
if not isinstance(getattr(self, attr, None), self.required_attributes[attr]):
raise errors.ActivityStreamValidationError("required attribute " + attr + " of type "
+ str(self.required_attributes[attr]))
# validate that every dictionary stored in object has @id and @type
self._validate_type_id_defined(self.__getattribute__("object"))
class Add(Activity):
type = "Add"
attributes = Activity.attributes + ["target"]
required_attributes = {**Activity.required_attributes, "target": dict}
class Remove(Activity):
type = "Remove"
attributes = Activity.attributes + ["target", "origin"]
def validate(self):
super().validate()
if not getattr(self, "target", None) and not getattr(self, "origin", None):
raise errors.ActivityStreamValidationError("Invalid activity, no target or origin given")
if getattr(self, "target", None) is not None:
if not isinstance(self.target, dict):
raise errors.ActivityStreamValidationError("Invalid target type, must be a dict")
if getattr(self, "origin", None) is not None:
if not isinstance(self.origin, dict):
raise errors.ActivityStreamValidationError("Invalid origin type, must be a dict")
class Create(Activity):
type = "Create"
class Update(Activity):
type = "Update"
class Delete(Activity):
type = "Delete"
attributes = Activity.attributes + ["origin"]
class Follow(Activity):
type = "Follow"
def validate(self):
super().validate()
if isinstance(self.actor, Actor) and (self.actor.inbox is None and self.actor.id is None):
raise errors.ActivityStreamValidationError("Must pass inbox or id with the actor to follow")
ALLOWED_TYPES.update({
"Activity": Activity,
"Add": Add,
"Remove": Remove,
"Create": Create,
"Update": Update,
"Delete": Delete,
"Follow": Follow
})
djangoldp/admin.py
View file @ e97d33ad
from importlib import import_module
from django.conf import settings
from csv import DictWriter
from django.contrib import admin
from .models import LDPSource, Model
from django.contrib.auth.admin import UserAdmin
from django.core.exceptions import FieldDoesNotExist
from django.http import HttpResponse
from guardian.admin import GuardedModelAdmin
from djangoldp.models import Activity, ScheduledActivity, Follower
from djangoldp.activities.services import ActivityQueueService
class DjangoLDPAdmin(GuardedModelAdmin):
'''
An admin model representing a federated object. Inherits from GuardedModelAdmin to provide Django-Guardian
object-level permissions
'''
actions = ['export_csv']
export_fields = []
def resolve_verbose_name(self, field_path):
field = self
for field_name in field_path.split('__'):
try:
field = field.model._meta.get_field(field_name)
except FieldDoesNotExist:
return None
return field.verbose_name
@admin.action(description="Export CSV")
def export_csv(self, request, queryset):
response = HttpResponse(content_type="text/csv")
response['Content-Disposition'] = f'attachment; filename="{self.model.__name__}.csv"'
# only keep fields that can be resolved, keep only urlid if none
field_list = list(filter(self.resolve_verbose_name, self.export_fields or self.list_display)) or ['urlid']
headers = {field:self.resolve_verbose_name(field) for field in field_list}
writer = DictWriter(response, fieldnames=field_list)
writer.writerow(headers)
writer.writerows(queryset.values(*field_list))
return response
class DjangoLDPUserAdmin(UserAdmin, DjangoLDPAdmin):
'''An extension of UserAdmin providing the functionality of DjangoLDPAdmin'''
list_display = ('urlid', 'email', 'first_name', 'last_name', 'date_joined', 'last_login', 'is_staff')
search_fields = ['urlid', 'email', 'first_name', 'last_name']
ordering = ['urlid']
def get_fieldsets(self, request, obj=None):
fieldsets = super().get_fieldsets(request, obj)
federated_fields = ['urlid', 'allow_create_backlink']
if self.exclude is not None:
federated_fields = list(set(federated_fields) - set(self.exclude))
for fieldset in fieldsets:
federated_fields = list(set(federated_fields) - set(fieldset[1]['fields']))
if len(federated_fields) == 0:
return fieldsets
fieldsets = [('Federation', {'fields': federated_fields})] + list(fieldsets)
return fieldsets
@admin.action(description='Resend activity')
def resend_activity(modeladmin, request, queryset):
for a in queryset:
ActivityQueueService.send_activity(a.external_id, a.to_activitystream())
resend_activity.short_description = 'Resend activity'
@admin.register(Activity, ScheduledActivity)
class ActivityAdmin(DjangoLDPAdmin):
fields = ['urlid', 'type', 'local_id', 'external_id', 'created_at', 'success', 'payload_view', 'response_code',
'response_location', 'response_body_view']
list_display = ['created_at', 'type', 'local_id', 'external_id', 'success', 'response_code']
readonly_fields = ['created_at', 'payload_view', 'response_location', 'response_code', 'response_body_view']
search_fields = ['urlid', 'type', 'local_id', 'external_id', 'response_code']
actions = [resend_activity]
def payload_view(self, obj):
return str(obj.to_activitystream())
for package in settings.DJANGOLDP_PACKAGES:
try:
import_module('{}.admin'.format(package))
except ModuleNotFoundError:
pass
def response_body_view(self, obj):
return str(obj.response_to_json())
for package in settings.DJANGOLDP_PACKAGES:
try:
import_module('{}.models'.format(package))
except ModuleNotFoundError:
pass
model_classes = {cls.__name__: cls for cls in Model.__subclasses__()}
@admin.register(Follower)
class FollowerAdmin(DjangoLDPAdmin):
fields = ['urlid', 'object', 'inbox', 'follower']
list_display = ['urlid', 'object', 'inbox', 'follower']
search_fields = ['object', 'inbox', 'follower']
for class_name in model_classes:
model_class = model_classes[class_name]
if not admin.site.is_registered(model_class):
admin.site.register(model_class)
admin.site.register(LDPSource)
djangoldp/apps.py
View file @ e97d33ad
import os
from django.apps import AppConfig
class DjangoldpConfig(AppConfig):
name = 'djangoldp'
def ready(self):
self.auto_register_model_admin()
self.start_activity_queue()
# Patch guardian core to avoid prefetching permissions several times
from guardian.core import ObjectPermissionChecker
ObjectPermissionChecker._prefetch_cache_orig = ObjectPermissionChecker._prefetch_cache
def _prefetch_cache(self):
if hasattr(self.user, "_guardian_perms_cache"):
self._obj_perms_cache = self.user._guardian_perms_cache
return
self._prefetch_cache_orig()
ObjectPermissionChecker._prefetch_cache = _prefetch_cache
def start_activity_queue(self):
from djangoldp.activities.services import ActivityQueueService
if os.environ.get('RUN_MAIN') is not None:
ActivityQueueService.start()
def auto_register_model_admin(self):
'''
Automatically registers Model subclasses in the admin panel (which have not already been added manually)
'''
from importlib import import_module
from django.conf import settings
from django.contrib import admin
from djangoldp.admin import DjangoLDPAdmin
from djangoldp.urls import get_all_non_abstract_subclasses
from djangoldp.models import Model
for package in settings.DJANGOLDP_PACKAGES:
try:
import_module('{}.admin'.format(package))
except ModuleNotFoundError:
pass
for package in settings.DJANGOLDP_PACKAGES:
try:
import_module('{}.models'.format(package))
except ModuleNotFoundError:
pass
for model in get_all_non_abstract_subclasses(Model):
if not admin.site.is_registered(model):
admin.site.register(model, DjangoLDPAdmin)
djangoldp/check_integrity.py 0 → 100644
View file @ e97d33ad
'''
DjangoLDP Check Integrity
Usage `./manage.py check_integrity --help`
'''
from django.apps import apps
from django.conf import settings
from djangoldp.models import LDPSource
from urllib.parse import urlparse
import requests
# Helper command for argument type checking
def is_string(target):
if isinstance(target, str):
return target
return False
# Helper command to check status code of a target
def is_alive(target, status_code = 200):
return requests.get(target).status_code == status_code
# Add argument to the `check_integrity` command
def add_arguments(parser):
parser.add_argument(
"--ignore",
action="store",
default=False,
type=is_string,
help="Ignore any server, comma separated",
)
parser.add_argument(
"--ignore-faulted",
default=False,
nargs="?",
const=True,
help="Ignore eventual faulted",
)
parser.add_argument(
"--ignore-404",
default=False,
nargs="?",
const=True,
help="Ignore eventual 404",
)
parser.add_argument(
"--fix-faulted-resources",
default=False,
nargs="?",
const=True,
help="Fix faulted resources",
)
parser.add_argument(
"--fix-404-resources",
default=False,
nargs="?",
const=True,
help="Fix 404 resources",
)
parser.add_argument(
"--fix-offline-servers",
default=False,
nargs="?",
const=True,
help="Remove resources from offline servers",
)
# Define our own checks
def check_integrity(options):
models = apps.get_models()
ignored = set()
if(options["ignore"]):
for target in options["ignore"].split(","):
ignored.add(urlparse(target).netloc)
if(len(ignored) > 0):
print("Ignoring servers:")
for server in ignored:
print("- "+server)
resources = set()
resources_map = dict()
base_urls = set()
for model in models:
for obj in model.objects.all():
if hasattr(obj, "urlid"):
if(obj.urlid):
if(not obj.urlid.startswith(settings.BASE_URL)):
url = urlparse(obj.urlid).netloc
if(url not in ignored):
resources.add(obj.urlid)
resources_map[obj.urlid] = obj
base_urls.add(url)
if(len(base_urls) > 0):
print("Found "+str(len(resources_map))+" distant resources on "+str(len(models))+" models")
print("Servers that I have backlinks to:")
for server in base_urls:
print("- "+server)
else:
print("I don't have any backlink")
source_urls = set()
for source in LDPSource.objects.all():
source_urls.add(urlparse(source.urlid).netloc)
if(len(source_urls) > 0):
print("Servers that I'm allowed to get federated to:")
for server in source_urls:
print("- "+server)
else:
print("I'm not federated")
difference_servers = base_urls.difference(source_urls)
if(len(difference_servers) > 0):
print("Servers that I should not get aware of:")
for server in difference_servers:
print("- "+server)
# Handle faulted resources
if(not options["ignore_faulted"]):
faulted_resources = set()
for server in difference_servers:
for resource in resources:
if(urlparse(resource).netloc in server):
faulted_resources.add(resource)
if(len(faulted_resources) > 0):
print("Resources in fault:")
for resource in faulted_resources:
print("- "+resource)
else:
print("No resource are in fault")
if(options["fix_faulted_resources"]):
for resource in faulted_resources:
try:
resources_map[resource].delete()
except:
pass
print("Fixed faulted resources")
else:
print("Fix them with `./manage.py check_integrity --fix-faulted-resources`")
else:
print("I accept datas for every of those servers")
# Handle 404 resources
if(not options["ignore_404"]):
resources_404 = set()
resources_servers_offline = set()
for resource in resources:
try:
if(is_alive(resource, 404)):
resources_404.add(resource)
except:
resources_servers_offline.add(resource)
if(len(resources_404) > 0):
print("Faulted resources, 404:")
for resource in resources_404:
print("- "+resource)
if(options["fix_404_resources"]):
for resource in resources_404:
try:
resources_map[resource].delete()
except:
pass
print("Fixed 404 resources")
else:
print("Fix them with `./manage.py check_integrity --fix-404-resources`")
if(len(resources_servers_offline) > 0):
print("Faulted resources, servers offline:")
for resource in resources_servers_offline:
print("- "+resource)
if(options["fix_offline_servers"]):
for resource in resources_servers_offline:
try:
resources_map[resource].delete()
except:
pass
print("Fixed resources on offline servers")
else:
print("Fix them with `./manage.py check_integrity --fix-offline-servers`")
else:
print("No 404 in known resources")
\ No newline at end of file
djangoldp/cli.py 0 → 100644
View file @ e97d33ad
import click
import sys
import yaml
import subprocess
from pkg_resources import resource_filename
from pathlib import Path
from django.core import management
from django.core.management.base import CommandError
from . import __version__
# click entrypoint
@click.group()
@click.version_option(__version__)
def main():
"""DjangoLDP CLI"""
@main.command()
@click.argument('name', nargs=1, required=False)
def initserver(name):
"""Start a DjangoLDP server."""
try:
# use directly pwd
directory = Path.cwd()
if name:
# create a directory from project name in pwd
directory = Path.cwd() / name
directory.mkdir(parents=False, exist_ok=False)
# get the template path
template = resource_filename(__name__, 'conf/server_template')
# wrap the default django-admin startproject command
# this call import django settings and configure it
# see: https://docs.djangoproject.com/fr/2.2/topics/settings/#calling-django-setup-is-required-for-standalone-django-usage
# see: https://github.com/django/django/blob/stable/2.2.x/django/core/management/templates.py#L108
# fix: in 2.2 gabarit files options has been renamed: https://github.com/django/django/blob/stable/2.2.x/django/core/management/templates.py#L53
management.call_command('startproject', name, directory, template=template, files=['settings.yml'])
except FileExistsError:
click.echo(f'Error: the folder {directory} already exists')
sys.exit(1)
except CommandError as e:
click.echo(f'Error: {e}')
directory.rmdir()
sys.exit(1)
@main.command()
@click.argument('name', nargs=1)
def startpackage(name):
"""Start a DjangoLDP package."""
try:
# set directory
directory = Path.cwd() / name
# get the template path
template = resource_filename(__name__, 'conf/package_template')
# create dir
directory.mkdir(parents=False, exist_ok=False)
# wrap the default startapp command
management.call_command('startapp', name, directory, template=template)
except FileExistsError:
click.echo(f'Error: the folder {directory} already exists')
sys.exit(1)
except CommandError as e:
click.echo(f'Error: {e}')
sys.exit(1)
@main.command()
def install():
"""Install project dependencies."""
try:
# load dependencies from config file
path = Path.cwd() / 'settings.yml'
with open(path, 'r') as f:
dependencies = yaml.safe_load(f).get('dependencies', [])
# install them by calling pip command
cmd = [sys.executable, "-m", "pip", "install", "--upgrade"]
try:
cmd.extend(dependencies)
subprocess.run(cmd).check_returncode()
click.echo('Installation done!')
except TypeError:
click.echo('No dependency to install')
except FileNotFoundError:
click.echo('Config error: no settings.yml file in this directory')
sys.exit(1)
except subprocess.CalledProcessError as e:
click.echo(f'Installation error: {e}')
sys.exit(1)
@main.command()
@click.option('--with-admin', 'admin', help='Create an administrator user with email.')
@click.option('--email', help='Provide an email for administrator.')
@click.option('--with-dummy-admin', 'dummy_admin', is_flag=True, help='Create a default "admin" user.')
def configure(admin, dummy_admin, email):
"""Configure the project."""
try:
# shortcut to the djangoldp.management command
path = str(Path.cwd() / 'manage.py')
cmd = [sys.executable, path, 'configure']
if admin:
if not email:
click.echo('Error: missing email for admin user')
return
cmd.extend(['--with-admin', admin, '--email', email])
elif dummy_admin:
cmd.append('--with-dummy-admin')
subprocess.run(cmd).check_returncode()
click.echo('Configuration done!')
except subprocess.CalledProcessError as e:
click.echo(f'Configuration error: {e}')
sys.exit(1)
@main.command()
def runserver():
"""Run the Django embeded webserver."""
try:
# shortcut to the djangoldp.management command
path = str(Path.cwd() / 'manage.py')
cmd = [sys.executable, path, 'runserver', '0.0.0.0:8000']
subprocess.run(cmd).check_returncode()
except subprocess.CalledProcessError as e:
click.echo(f'Execution error: {e}')
sys.exit(1)
djangoldp/conf/__init__.py 0 → 100644
View file @ e97d33ad
djangoldp/conf/default_settings.py 0 → 100644
View file @ e97d33ad
"""This module override some of default django configuration from the global settings."""
from django.conf.global_settings import *
####################
# LDP #
####################
LDP_RDF_CONTEXT = 'https://cdn.startinblox.com/owl/context.jsonld'
MAX_ACTIVITY_RESCHEDULES = 3
DEFAULT_BACKOFF_FACTOR = 1
DEFAULT_ACTIVITY_DELAY = 0.2
DEFAULT_REQUEST_TIMEOUT = 10
LDP_INCLUDE_INNER_PERMS = False
####################
# CORE #
####################
# https://en.wikipedia.org/wiki/List_of_tz_zones_by_name (although not all
# systems may support all possibilities). When USE_TZ is True, this is
# interpreted as the default user time zone.
TIME_ZONE = 'UTC'
# If you set this to True, Django will use timezone-aware datetimes.
USE_TZ = True
# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'en-us'
# Database connection info. If left empty, will default to the dummy backend.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': 'db.sqlite3',
}
}
# List of strings representing installed apps.
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'djangoldp',
'guardian'
]
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
# Default server url
SITE_URL = 'http://localhost:8000'
BASE_URL = SITE_URL
# Default URL conf
ROOT_URLCONF = 'djangoldp.urls'
# Absolute filesystem path to the directory that will hold user-uploaded files.
# Example: "/var/www/example.com/media/"
MEDIA_ROOT = ''
# URL that handles the media served from MEDIA_ROOT.
# Examples: "http://example.com/media/", "http://media.example.com/"
MEDIA_URL = '/media/'
# Absolute path to the directory static files should be collected to.
# Example: "/var/www/example.com/static/"
STATIC_ROOT = None
# URL that handles the static files served from STATIC_ROOT.
# Example: "http://example.com/static/", "http://static.example.com/"
STATIC_URL = '/static/'
# ETAGS config
USE_ETAGS = True
# Default X-Frame-Options header value
X_FRAME_OPTIONS = 'DENY'
# The Python dotted path to the WSGI application that Django's internal server
# (runserver) will use. If `None`, the return value of
# 'django.core.wsgi.get_wsgi_application' is used, thus preserving the same
# behavior as previous versions of Django. Otherwise this should point to an
# actual WSGI application object.
WSGI_APPLICATION = 'server.wsgi.application'
##############
# MIDDLEWARE #
##############
# List of middleware to use. Order is important; in the request phase, these
# middleware will be applied in the order given, and in the response
# phase the middleware will be applied in reverse order.
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'djangoldp.middleware.AllowRequestedCORSMiddleware',
'django.middleware.gzip.GZipMiddleware',
'django_brotli.middleware.BrotliMiddleware'
]
##################
# REST FRAMEWORK #
##################
REST_FRAMEWORK = {
'DEFAULT_SCHEMA_CLASS': 'drf_spectacular.openapi.AutoSchema',
'DEFAULT_PAGINATION_CLASS': 'djangoldp.pagination.LDPPagination'
}
###################
# DRF SPECTACULAR #
###################
ENABLE_SWAGGER_DOCUMENTATION = False
SPECTACULAR_SETTINGS = {
'TITLE': 'DjangoLDP Based API Description',
'DESCRIPTION': 'Here you will find the list of all endpoints available on your Djangoldp-based server instance and\
the available methods, needed parameters and requests examples. The list of available endpoints depend on your instance configuration\
especially the list of bloxes and associated django models activated.',
'VERSION': '2.1.37',
'SERVE_INCLUDE_SCHEMA': False
}
############
# SESSIONS #
############
# Same site policy
DCS_SESSION_COOKIE_SAMESITE = 'none'
##################
# AUTHENTICATION #
##################
AUTHENTICATION_BACKENDS = ['django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend']
OIDC_ACCESS_CONTROL_ALLOW_HEADERS = 'Content-Type, if-match, accept, authorization, DPoP, cache-control, pragma, prefer'
# The minimum number of seconds a password reset link is valid for
PASSWORD_RESET_TIMEOUT = 60 * 60 * 24 * 3
DISABLE_LOCAL_OBJECT_FILTER = False
GUARDIAN_AUTO_PREFETCH = True
DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
djangoldp/conf/ldpsettings.py 0 → 100644
View file @ e97d33ad
import os
import sys
import yaml
import logging
from django.core.exceptions import ImproperlyConfigured
from django.conf import settings as django_settings
from pathlib import Path
from collections import OrderedDict
from typing import Iterable
from importlib import import_module
from . import default_settings
logger = logging.getLogger(__name__)
def configure(filename='settings.yml'):
"""Helper function to configure django from LDPSettings."""
yaml_config = None
try:
with open(filename, 'r') as f:
yaml_config = yaml.safe_load(f)
except FileNotFoundError:
logger.info('Starting project without configuration file')
# ref: https://docs.djangoproject.com/fr/2.2/topics/settings/#custom-default-settings
ldpsettings = LDPSettings(yaml_config)
django_settings.configure(ldpsettings)
class LDPSettings(object):
"""Class managing the DjangoLDP configuration."""
def __init__(self, config):
"""Build a Django Setting object from a dict."""
if django_settings.configured:
raise ImproperlyConfigured('Settings have been configured already')
self._config = config
self._explicit_settings = set(default_settings.__dict__.keys()) #default settings are explicit
self._settings = self.build_settings()
def build_settings(self, extend=['INSTALLED_APPS', 'MIDDLEWARE']):
"""
Look for the parameters in multiple places.
Each step overrides the value of the previous key found. Except for "extend" list. Those value must be lists and all values found are added to these lists without managing duplications.
Resolution order of the configuration:
1. Core default settings
2. Packages settings
3. Code from a local settings.py file
4. YAML config file
"""
# helper loop
def update_with(config):
for setting, value in config.items():
self._explicit_settings.add(setting)
if setting in extend:
settings[setting].extend(value)
elif not setting.startswith('_'):
settings.update({setting: value})
# start from default core settings
settings = default_settings.__dict__.copy()
logger.debug(f'Building settings from core defaults')
# INSTALLED_APPS starts empty
settings['INSTALLED_APPS'] = []
# look settings from packages in the order they are given (local overrides installed)
for pkg in self.DJANGOLDP_PACKAGES:
# FIXME: There is something better to do here with the sys.modules path
try:
# override with values from installed package
mod = import_module(f'{pkg}.djangoldp_settings')
update_with(mod.__dict__)
logger.debug(f'Updating settings from installed package {pkg}')
except ModuleNotFoundError:
pass
try:
# override with values from local package
mod = import_module(f'{pkg}.{pkg}.djangoldp_settings')
update_with(mod.__dict__)
logger.debug(f'Updating settings from local package {pkg}')
except ModuleNotFoundError:
pass
# look in settings.py file in directory
try:
mod = import_module('settings')
update_with(mod.__dict__)
logger.debug(f'Updating settings from local settings.py file')
except ModuleNotFoundError:
pass
# look in YAML config file 'server' section
try:
conf = self._config.get('server', {})
update_with(conf)
logger.debug(f'Updating settings with project config')
except KeyError:
pass
# In the end adds the INSTALLED_APPS from the core
settings['INSTALLED_APPS'].extend(getattr(default_settings,'INSTALLED_APPS'))
return settings
@property
def DJANGOLDP_PACKAGES(self):
"""Returns the list of LDP packages configured."""
pkg = self._config.get('ldppackages', [])
return [] if pkg is None else pkg
@property
def INSTALLED_APPS(self):
"""Return the installed apps and the LDP packages."""
# get ldp packages (they are django apps)
apps = self.DJANGOLDP_PACKAGES.copy()
# add the default apps
apps.extend(self._settings['INSTALLED_APPS'])
# As settings come from different origins duplicuation is likeliy to happen
return list(OrderedDict.fromkeys(apps))
def __getattr__(self, param):
"""Return the requested parameter from cached settings."""
if param.startswith('_') or param.islower():
# raise the django exception for inexistent parameter
raise AttributeError(f'"{param}" is not compliant to django settings format')
try:
return self._settings[param]
except KeyError:
# raise the django exception for inexistent parameter
raise AttributeError(f'no "{param}" parameter found in settings')
def is_overridden(self, setting):
return setting in self._explicit_settings
\ No newline at end of file