Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • djangoldp-packages/djangoldp
  • decentral1se/djangoldp
  • femmefaytale/djangoldp
  • jvtrudel/djangoldp
4 results
Show changes
Commits on Source (565)
465 additional commits have been omitted to prevent performance issues.
Expand all Hide whitespace changes
Inline Side-by-side
Showing
with 1252 additions and 1270 deletions
.gitignore
View file @ e97d33ad
......@@ -5,3 +5,7 @@ build
*.pyc
*~
*.swp
djangoldp/tests/tests_temp.py
*/.idea/*
.DS_STORE
venv
.gitlab-ci.yml
View file @ e97d33ad
---
image: python:3.6
image: python:3.11
stages:
- test
- release
include:
project: infra/gitlab
ref: master
file: templates/python.ci.yml
test:
stage: test
......@@ -11,22 +12,22 @@ test:
- pip install .[dev]
- python -m unittest djangoldp.tests.runner
except:
- master
- tags
tags:
- test
publish:
stage: release
before_script:
- pip install python-semantic-release~=5.0 sib-commit-parser~=0.3
- git config user.name "${GITLAB_USER_NAME}"
- git config user.email "${GITLAB_USER_EMAIL}"
- git remote set-url origin "https://gitlab-ci-token:${GL_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git"
- git fetch --tags
crypto-test:
stage: test
script:
- semantic-release publish
only:
- master
- pip install .[crypto]
- python -m unittest djangoldp_crypto.tests.runner
except:
- tags
tags:
- deploy
- test
publish:
cache: []
extends: .publish_pypi
CHANGELOG.md
View file @ e97d33ad
# Changelog
<!--next-version-placeholder-->
## v3.0.5 (2023-07-24)
### Fix
* Readme ([`8792530`](https://github.com/djangoldp-packages/djangoldp/commit/87925305b2093282230511bceb38978db4b279a2))
## v3.0.4 (2023-07-24)
### Fix
* Readme ([`eeedc33`](https://github.com/djangoldp-packages/djangoldp/commit/eeedc3378ed3f4e454377431da6bb50202efdcdc))
## v3.0.3 (2023-07-24)
### Fix
* Readme ([`73597b6`](https://github.com/djangoldp-packages/djangoldp/commit/73597b65430a4d23306f78def0331bda60857493))
## Unreleased
* Imported CLI along with development template from sib-manager project
Dockerfile 0 → 100644
View file @ e97d33ad
# use with your own settings.yml
FROM python:3.11
LABEL maintainer="Plup <plup@plup.io>"
# get server
RUN pip install djangoldp
# create a server instance
RUN djangoldp initserver ldpserver
WORKDIR /ldpserver
#COPY settings.yml .
RUN djangoldp install
RUN djangoldp configure --with-dummy-admin
# run the server
EXPOSE 8000
CMD ["djangoldp", "runserver"]
LICENSE
View file @ e97d33ad
MIT License
Copyright (c) 2018 Startin blox
Copyright (c) 2018-2023 Startin blox
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
......
MANIFEST.in
View file @ e97d33ad
graft djangoldp/conf/package_template
graft djangoldp/conf/server_template
graft djangoldp/templates/
README.md
View file @ e97d33ad
## Synopsis
This module is an add-on for Django REST Framework that serves a django model respecting the Linked Data Platform convention.
It aims at enabling people with little development skills to serve their own data, to be used with a LDP application.
Building a Startin' Blox application? Read this: https://git.happy-dev.fr/startinblox/devops/doc
## Requirements
* Django (known to work with django 1.11)
* Django Rest Framework
* pyld==1.0.5
* django-guardian
* djangorestframework-guardian
# Setup a DjangoLDP server
## Installation
Check the [official documentation](https://docs.startinblox.com/import_documentation/djangoldp_guide/install-djangoldp-server.html).
1. Install this module and all its dependencies
## Settings
```bash
$ pip install djangoldp
```
2. Create a django project
```bash
$ django-admin startproject myldpserver
```
3. Add DjangoLDP to INSTALLED_APPS
```python
INSTALLED_APPS = [
...
# make sure all of your own apps are installed BEFORE DjangoLDP
'djangoldp.apps.DjangoldpConfig',
]
```
IMPORTANT: DjangoLDP will register any models which haven't been registered, with the admin. As such it is important to add your own apps above DjangoLDP, so that you can use custom Admin classes if you wish
### User model requirements
When implementing authentication in your own application, you have two options:
* Using or extending [DjangoLDP-Account](https://git.startinblox.com/djangoldp-packages/djangoldp-account), a DjangoLDP package modelling federated users
* Using your own user model & defining the authentication behaviour yourself
Please see the [Authentication guide](https://git.startinblox.com/djangoldp-packages/djangoldp/wikis/guides/authentication) for full information
If you're going to use your own model then for federated login to work your user model must extend `DjangoLDP.Model`, or define a `urlid` field on the user model, for example:
```python
urlid = LDPUrlField(blank=True, null=True, unique=True)
```
If you don't include this field, then all users will be treated as users local to your instance
The `urlid` field is used to uniquely identify the user and is part of the Linked Data Protocol standard. For local users it can be generated at runtime, but for some resources which are from distant servers this is required to be stored
## Creating your first model
1. Create your django model inside a file myldpserver/myldpserver/models.py
Note that container_path will be use to resolve instance iri and container iri
In the future it could also be used to auto configure django router (e.g. urls.py)
```python
from djangoldp.models import Model
* `OIDC_ACCESS_CONTROL_ALLOW_HEADERS`: overrides the access control headers allowed in the `Access-Control-Allow-Headers` CORS header of responses. Defaults to `authorization, Content-Type, if-match, accept, DPoP`
* `ANONYMOUS_USER_NAME` a setting inherited from dependency [Django-Guardian](https://django-guardian.readthedocs.io/en/stable/overview.html)
* `DJANGOLDP_PERMISSIONS`: overrides the list of all permissions on all resources
* `SERIALIZER_CACHE`: toggles the use of a built-in cache in the serialization of containers/resources
* `MAX_RECORDS_SERIALIZER_CACHE`: sets the maximum number of serializer cache records, at which point the cache will be cleared (reset). Defaults to 10,000
* `SEND_BACKLINKS`: enables the searching and sending of [Activities](https://git.startinblox.com/djangoldp-packages/djangoldp/-/wikis/guides/federation) to distant resources linked by users to this server
* `MAX_ACTIVITY_RESCHEDULES`, `DEFAULT_BACKOFF_FACTOR`, `DEFAULT_ACTIVITY_DELAY`, `DEFAULT_REQUEST_TIMEOUT` tweaks the behaviour of the ActivityQueueService
* `STORE_ACTIVITIES`: sets whether to store activities sent and backlinks received, or to treat them as transient (value should be `"VERBOSE"`, `"ERROR"` or `None`). Defaults to `"ERROR"`
* `MAX_RECORDS_ACTIVITY_CACHE`: sets the maximum number of serializer cache records, at which point the cache will be cleared (reset). If set to 0 disables the cache. Defaults to 10,000
* `ENABLE_SWAGGER_DOCUMENTATION`: enables the automatic OpenAPI-based API schema and documentation generation, made available at `http://yourserver/docs/` is the flag is set to True. Default to False
* `DISABLE_LOCAL_OBJECT_FILTER`: disabled the LocalObjectBackendFilter which is processing-time costly and only need activation in federated architecture, so we preferred to add a way to disable it as a workaround for in-progress performances improvements. Default to False
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
```
1.1. Configure container path (optional)
By default it will be "todos/" with an S for model called Todo
```python
<Model>._meta.container_path = "/my-path/"
```
1.2. Configure field visibility (optional)
Note that at this stage you can limit access to certain fields of models using
```python
<Model>._meta.serializer_fields (<>list of field names to show>)
```
For example, if you have a model with a related field with type **django.contrib.auth.models.User** you don't want to show personal details or password hashes.
E.g.
```python
from django.contrib.auth.models import User
User._meta.serializer_fields = ('username','first_name','last_name')
```
Note that this will be overridden if you explicitly set the fields= parameter as an argument to LDPViewSet.urls(), and filtered if you set the excludes= parameter.
2. Add a url in your urls.py:
```python
from django.conf.urls import url
from django.contrib import admin
from djangoldp.views import LDPViewSet
from .models import Todo
urlpatterns = [
url(r'^', include('djangoldp.urls')),
url(r'^admin/', admin.site.urls), # Optional
]
```
This creates 2 routes for each Model, one for the list, and one with an ID listing the detail of an object.
You could also only use this line in settings.py instead:
```python
ROOT_URLCONF = 'djangoldp.urls'
```
## Synopsis
3. In the settings.py file, add your application name at the beginning of the application list, and add the following lines
This module is an add-on for Django REST Framework that serves a django model respecting the Linked Data Platform convention.
```python
STATIC_ROOT = os.path.join(os.path.dirname(BASE_DIR), 'static')
LDP_RDF_CONTEXT = 'https://cdn.happy-dev.fr/owl/hdcontext.jsonld'
DJANGOLDP_PACKAGES = []
SITE_URL = 'http://localhost:8000'
BASE_URL = SITE_URL
```
It aims at enabling people with little development skills to serve their own data, to be used with a LDP application.
* `LDP_RDF_CONTEXT` tells DjangoLDP where our RDF [ontology](https://www.w3.org/standards/semanticweb/ontology) is defined, which will be returned as part of our views in the 'context' field. This is a web URL and you can visit the value to view the full ontology online. The ontology can be a string, as in the example, but it can also be a dictionary, or a list of ontologies (see the [JSON-LD spec](https://json-ld.org) for examples)
* `DJANGOLDP_PACKAGES` defines which other [DjangoLDP packages](https://git.happy-dev.fr/startinblox/djangoldp-packages) we're using in this installation
* `SITE_URL` is the URL serving the site, e.g. `https://example.com/`. Note that if you include the DjangoLDP urls in a nested path (e.g. `https://example.com/api/`), then `SITE_URL` will need to be set to this value
* `BASE_URL` may be different from SITE_URL, e.g. `https://example.com/app/`
## Check technical documentation
* [Using DjangoLDP with your models](./docs/create_model.md)
4. You can also register your model for the django administration site
## Contribute to DjangoLDP
```python
from django.contrib import admin
from djangoldp.admin import DjangoLDPAdmin
from .models import Todo
### Testing
admin.site.register(Todo, DjangoLDPAdmin)
```
Packaged with DjangoLDP is a tests module, containing unit tests
5. You then need to have your WSGI server pointing on myldpserver/myldpserver/wsgi.py
You can extend these tests and add your own test cases by following the examples in the code. You can then run your tests with:
`python -m unittest djangoldp.tests.runner`
6. You will probably need to create a super user
## Check your datas integrity
```bash
$ ./manage.py createsuperuser
```
Because of the way the DjangoLDP's federation work, you can reach some integrity issue within your datas.
7. If you have no CSS on the admin screens :
You can check them with:
```bash
$ ./manage.py collectstatic
./manage.py check_integrity
```
## Execution
To start the server, `cd` to the root of your Django project and run :
You can ignore some servers:
```bash
$ python3 manage.py runserver
./manage.py check_integrity --ignore "https://server/,https://another-server/"
```
## Using DjangoLDP
### Models
To use DjangoLDP in your models you just need to extend djangoldp.Model
The Model class allows you to use your models in federation, adding a `urlid` field, and some key methods useful in federation
### Add you own commands to the `check_integrity` from your own package
If you define a Meta for your Model, you will [need to explicitly inherit Model.Meta](https://docs.djangoproject.com/fr/2.2/topics/db/models/#meta-inheritance) in order to inherit the default settings, e.g. `default_permissions`
Create a `check_integrity.py` file within your app folder containing:
```python
from djangoldp.models import Model, LDPMetaMixin
def add_arguments(parser):
parser.add_argument(
"--my-own-argument",
default=False,
nargs="?",
const=True,
help="Some help text",
)
class Todo(Model):
name = models.CharField(max_length=255)
class Meta(Model.Meta):
```
See "Custom Meta options" below to see some helpful ways you can tweak the behaviour of DjangoLDP
Your model will be automatically detected and registered with an LDPViewSet and corresponding URLs, as well as being registered with the Django admin panel. If you register your model with the admin panel manually, make sure to extend djangoldp.DjangoLDPAdmin so that the model is registered with [Django-Guardian object permissions](https://django-guardian.readthedocs.io/en/stable/userguide/admin-integration.html). An alternative version which extends Django's `UserAdmin` is available as djangoldp.DjangoLDPUserAdmin
#### Model Federation
Model `urlid`s can be **local** (matching `settings.SITE_URL`), or **external**
To maintain consistency between federated servers, [Activities](https://www.w3.org/TR/activitystreams-vocabulary) such as Create, Update, Delete are sent to external resources referenced in a ForeignKey relation, instructing them on how to manage the reverse-links with the local server
This behaviour can be disabled in settings.py
```python
SEND_BACKLINKS = False
def check_integrity(options):
if(options["my_own_argument"]):
print("You ran a check_integrity with --my-own-argument!")
else:
print("Run me with `./manage.py check_integrity --my-own-argument`")
```
It can also be disabled on a model instance
```python
instance.allow_create_backlinks = False
```
### LDPManager
DjangoLDP Models override `models.Manager`, accessible by `Model.objects`
#### local()
For situations where you don't want to include federated resources in a queryset e.g.
```python
Todo.objects.create(name='Local Todo')
Todo.objects.create(name='Distant Todo', urlid='https://anotherserversomewhere.com/todos/1/')
Todo.objects.all() # query set containing { Local Todo, Distant Todo }
Todo.objects.local() # { Local Todo } only
```
For Views, we also define a FilterBackend to achieve the same purpose. See the section on ViewSets for this purpose
#### nested_fields()
returns a list of all nested field names for the model, built of a union of the model class' `nested_fields` setting, the to-many relations on the model, excluding all fields detailed by `nested_fields_exclude`
## LDPViewSet
DjangoLDP automatically generates ViewSets for your models, and registers these at urls, according to the settings configured in the model Meta (see below for options)
### Custom Parameters
#### lookup_field
Can be used to use a slug in the url instead of the primary key.
```python
LDPViewSet.urls(model=User, lookup_field='username')
```
#### nested_fields
list of ForeignKey, ManyToManyField, OneToOneField and their reverse relations. When a field is listed in this parameter, a container will be created inside each single element of the container.
In the following example, besides the urls `/members/` and `/members/<pk>/`, two other will be added to serve a container of the skills of the member: `/members/<pk>/skills/` and `/members/<pk>/skills/<pk>/`
```python
<Model>._meta.nested_fields=["skills"]
```
## Filter Backends
To achieve federation, DjangoLDP includes links to objects from federated servers and stores these as local objects (see 1.0 - Models). In some situations, you will want to exclude these from the queryset of a custom view
To provide for this need, there is defined in `djangoldp.filters` a FilterBackend which can be included in custom viewsets to restrict the queryset to only objects which were created locally:
```python
from djangoldp.filters import LocalObjectFilterBackend
class MyViewSet(..):
filter_backends=[LocalObjectFilterBackend]
```
By default, LDPViewset applies filter backends from the `permission_classes` defined on the model (see 3.1 for configuration)
By default, `LDPViewSets` use another FilterBackend, `LocalObjectOnContainerPathBackend`, which ensures that only local objects are returned when the path matches that of the Models `container_path` (e.g. /users/ will return a list of local users). In very rare situations where this might be undesirable, it's possible to extend `LDPViewSet` and remove the filter_backend:
```python
class LDPSourceViewSet(LDPViewSet):
model = LDPSource
filter_backends = []
```
Following this you will need to update the model's Meta to use the custom `view_set`:
```python
class Meta:
view_set = LDPSourceViewSet
```
## Custom Meta options on models
### rdf_type
Indicates the type the model corresponds to in the ontology. E.g. where `'hd:circle'` is defined in an ontology from `settings.LDP_RDF_CONTEXT`
```python
rdf_type = 'hd:circle'
```
### rdf_context
Sets added `context` fields to be serialized with model instances
```python
rdf_context = {'picture': 'foaf:depiction'}
```
### auto_author
This property allows to associate a model with the logged in user.
```python
class MyModel(models.Model):
author_user = models.ForeignKey(settings.AUTH_USER_MODEL)
class Meta:
auto_author = 'author_user'
```
Now when an instance of `MyModel` is saved, its `author_user` property will be set to the authenticated user.
### auto_author_field
Set this property to make the value of the `auto_author` field a property on the authenticated use.
```python
class MyModel(models.Model):
author_user = models.ForeignKey(settings.AUTH_USER_MODEL)
class Meta:
auto_author = 'author_user'
auto_author_field = 'profile'
```
Now when an instance of `MyModel` is saved, its `author_user` property will be set to the **profile** of the authenticated user.
## permissions
Django-Guardian is used by default to support object-level permissions. Custom permissions can be added to your model using this attribute. See the [Django-Guardian documentation](https://django-guardian.readthedocs.io/en/stable/userguide/assign.html) for more information
## permissions_classes
This allows you to add permissions for anonymous, logged in user, author ... in the url:
By default `LDPPermissions` is used.
Specific permissin classes can be developed to fit special needs.
## anonymous_perms, user_perms, owner_perms
Those allow you to set permissions from your model's meta.
You can give the following permission to them:
* `view`
* `add`
* `change`
* `control`
* `delete`
* `inherit`
With inherit, Users can herit from Anons. Also Owners can herit from Users.
Eg. with this model Anons can view, Auths can add & Owners can edit & delete.
Note that `owner_perms` need a `owner_field` meta that point the field with owner user.
```python
from djangoldp.models import Model
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
user = models.ForeignKey(settings.AUTH_USER_MODEL)
class Meta:
anonymous_perms = ['view']
authenticated_perms = ['inherit', 'add']
owner_perms = ['inherit', 'change', 'control', 'delete']
owner_field = 'user'
```
Important note:
If you need to give permissions to owner's object, don't forget to add auto_author in model's meta
### view_set
In case of custom viewset, you can use
```python
from djangoldp.models import Model
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
class Meta:
view_set = TodoViewSet
```
### container_path
See 3.1. Configure container path (optional)
### serializer_fields
```python
from djangoldp.models import Model
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
class Meta:
serializer_fields = ['name']
```
Only `name` will be serialized
### serializer_fields_exclude
```python
from djangoldp.models import Model
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
class Meta:
serializer_fields_exclude = ['name']
```
Only `deadline` will be serialized
This is achieved when `LDPViewSet` sets the `exclude` property on the serializer in `build_serializer` method. Note that if you use a custom viewset which does not extend LDPSerializer then you will need to set this property yourself
### nested_fields -- DEPRECIATED
Set on a model to auto-generate viewsets and containers for nested relations (e.g. `/circles/<pk>/members/`)
Depreciated in DjangoLDP 0.8.0, as all to-many fields are included as nested fields by default
### nested_fields_exclude
```python
<Model>._meta.nested_fields_exclude=["skills"]
```
Will exclude the field `skills` from the model's nested fields, and prevent a container `/model/<pk>/skills/` from being generated
## Custom urls
To add customs urls who can not be add through the `Model` class, it's possible de create a file named `djangoldp_urls.py`. It will be executed like an `urls.py` file
## Pagination
To enable pagination feature just add this configuration to the server `settings.py` :
```python
REST_FRAMEWORK = {
'DEFAULT_PAGINATION_CLASS': 'djangoldp.pagination.LDPPagination',
'PAGE_SIZE': 20
}
```
## Sources
To enable sources auto creation for all models, change `djangoldp` by `djangoldp.apps.DjangoldpConfig`, on `INSTALLED_APPS`
```python
INSTALLED_APPS = [
'djangoldp.apps.DjangoldpConfig',
]
```
## 301 on domain mismatch
To enable 301 redirection on domain mismatch, add `djangoldp.middleware.AllowOnlySiteUrl` on `MIDDLEWARE`
This ensure that your clients will use `SITE_URL` and avoid mismatch betwen url & the id of a resource/container
```python
MIDDLEWARE = [
'djangoldp.middleware.AllowOnlySiteUrl',
]
```
Notice tht it'll redirect only HTTP 200 Code.
## Extending DjangoLDP
### Testing
Packaged with DjangoLDP is a tests module, containing unit tests
You can extend these tests and add your own test cases by following the examples in the code. You can then run your tests with:
`python -m unittest djangoldp.tests.runner`
You can see a sample on the `check_integrity.py` file of DjangoLDP.
## License
......
djangoldp/__init__.py
View file @ e97d33ad
......@@ -3,7 +3,6 @@ from django.db.models import options
__version__ = '0.0.0'
options.DEFAULT_NAMES += (
'lookup_field', 'rdf_type', 'rdf_context', 'auto_author', 'auto_author_field', 'owner_field', 'view_set',
'container_path', 'permission_classes', 'serializer_fields', 'serializer_fields_exclude', 'nested_fields',
'nested_fields_exclude', 'depth', 'anonymous_perms', 'authenticated_perms', 'owner_perms')
default_app_config = 'djangoldp.apps.DjangoldpConfig'
'lookup_field', 'rdf_type', 'rdf_context', 'auto_author', 'owner_field', 'owner_urlid_field',
'view_set', 'container_path', 'permission_classes', 'serializer_fields', 'serializer_fields_exclude', 'empty_containers',
'nested_fields', 'depth', 'permission_roles', 'inherit_permissions', 'public_field', 'static_version', 'static_params', 'active_field', 'disable_url')
djangoldp/activities/services.py
View file @ e97d33ad
This diff is collapsed.
djangoldp/activities/verbs.py
View file @ e97d33ad
......@@ -33,12 +33,24 @@ class Activity(Object):
delattr(new, "bcc")
return new
def _validate_type_id_defined(self, value):
'''recursively ensures that all nested dict items define @id and @type attributes'''
for item in value.items():
if isinstance(item[1], dict):
item_value = item[1]
if '@type' not in item_value or '@id' not in item_value:
raise errors.ActivityStreamValidationError("all sub-objects passed in activity object must define @id and @type tags")
self._validate_type_id_defined(item_value)
def validate(self):
for attr in self.required_attributes.keys():
if not isinstance(getattr(self, attr, None), self.required_attributes[attr]):
raise errors.ActivityStreamValidationError("required attribute " + attr + " of type "
+ str(self.required_attributes[attr]))
# validate that every dictionary stored in object has @id and @type
self._validate_type_id_defined(self.__getattribute__("object"))
class Add(Activity):
type = "Add"
......
djangoldp/admin.py
View file @ e97d33ad
from guardian.admin import GuardedModelAdmin
from csv import DictWriter
from django.contrib import admin
from django.contrib.auth.admin import UserAdmin
from django.core.exceptions import FieldDoesNotExist
from django.http import HttpResponse
from guardian.admin import GuardedModelAdmin
from djangoldp.models import Activity, ScheduledActivity, Follower
from djangoldp.activities.services import ActivityQueueService
class DjangoLDPAdmin(GuardedModelAdmin):
......@@ -7,13 +13,38 @@ class DjangoLDPAdmin(GuardedModelAdmin):
An admin model representing a federated object. Inherits from GuardedModelAdmin to provide Django-Guardian
object-level permissions
'''
pass
actions = ['export_csv']
export_fields = []
def resolve_verbose_name(self, field_path):
field = self
for field_name in field_path.split('__'):
try:
field = field.model._meta.get_field(field_name)
except FieldDoesNotExist:
return None
return field.verbose_name
@admin.action(description="Export CSV")
def export_csv(self, request, queryset):
response = HttpResponse(content_type="text/csv")
response['Content-Disposition'] = f'attachment; filename="{self.model.__name__}.csv"'
# only keep fields that can be resolved, keep only urlid if none
field_list = list(filter(self.resolve_verbose_name, self.export_fields or self.list_display)) or ['urlid']
headers = {field:self.resolve_verbose_name(field) for field in field_list}
writer = DictWriter(response, fieldnames=field_list)
writer.writerow(headers)
writer.writerows(queryset.values(*field_list))
return response
class DjangoLDPUserAdmin(UserAdmin, GuardedModelAdmin):
class DjangoLDPUserAdmin(UserAdmin, DjangoLDPAdmin):
'''An extension of UserAdmin providing the functionality of DjangoLDPAdmin'''
list_display = ('urlid', 'email', 'first_name', 'last_name', 'is_staff')
list_display = ('urlid', 'email', 'first_name', 'last_name', 'date_joined', 'last_login', 'is_staff')
search_fields = ['urlid', 'email', 'first_name', 'last_name']
ordering = ['urlid']
def get_fieldsets(self, request, obj=None):
fieldsets = super().get_fieldsets(request, obj)
......@@ -31,3 +62,35 @@ class DjangoLDPUserAdmin(UserAdmin, GuardedModelAdmin):
fieldsets = [('Federation', {'fields': federated_fields})] + list(fieldsets)
return fieldsets
@admin.action(description='Resend activity')
def resend_activity(modeladmin, request, queryset):
for a in queryset:
ActivityQueueService.send_activity(a.external_id, a.to_activitystream())
resend_activity.short_description = 'Resend activity'
@admin.register(Activity, ScheduledActivity)
class ActivityAdmin(DjangoLDPAdmin):
fields = ['urlid', 'type', 'local_id', 'external_id', 'created_at', 'success', 'payload_view', 'response_code',
'response_location', 'response_body_view']
list_display = ['created_at', 'type', 'local_id', 'external_id', 'success', 'response_code']
readonly_fields = ['created_at', 'payload_view', 'response_location', 'response_code', 'response_body_view']
search_fields = ['urlid', 'type', 'local_id', 'external_id', 'response_code']
actions = [resend_activity]
def payload_view(self, obj):
return str(obj.to_activitystream())
def response_body_view(self, obj):
return str(obj.response_to_json())
@admin.register(Follower)
class FollowerAdmin(DjangoLDPAdmin):
fields = ['urlid', 'object', 'inbox', 'follower']
list_display = ['urlid', 'object', 'inbox', 'follower']
search_fields = ['object', 'inbox', 'follower']
djangoldp/apps.py
View file @ e97d33ad
import os
from django.apps import AppConfig
class DjangoldpConfig(AppConfig):
......@@ -5,6 +6,24 @@ class DjangoldpConfig(AppConfig):
def ready(self):
self.auto_register_model_admin()
self.start_activity_queue()
# Patch guardian core to avoid prefetching permissions several times
from guardian.core import ObjectPermissionChecker
ObjectPermissionChecker._prefetch_cache_orig = ObjectPermissionChecker._prefetch_cache
def _prefetch_cache(self):
if hasattr(self.user, "_guardian_perms_cache"):
self._obj_perms_cache = self.user._guardian_perms_cache
return
self._prefetch_cache_orig()
ObjectPermissionChecker._prefetch_cache = _prefetch_cache
def start_activity_queue(self):
from djangoldp.activities.services import ActivityQueueService
if os.environ.get('RUN_MAIN') is not None:
ActivityQueueService.start()
def auto_register_model_admin(self):
'''
......@@ -15,6 +34,7 @@ class DjangoldpConfig(AppConfig):
from django.conf import settings
from django.contrib import admin
from djangoldp.admin import DjangoLDPAdmin
from djangoldp.urls import get_all_non_abstract_subclasses
from djangoldp.models import Model
for package in settings.DJANGOLDP_PACKAGES:
......@@ -29,9 +49,6 @@ class DjangoldpConfig(AppConfig):
except ModuleNotFoundError:
pass
model_classes = {cls.__name__: cls for cls in Model.__subclasses__()}
for class_name in model_classes:
model_class = model_classes[class_name]
if not admin.site.is_registered(model_class):
admin.site.register(model_class, DjangoLDPAdmin)
for model in get_all_non_abstract_subclasses(Model):
if not admin.site.is_registered(model):
admin.site.register(model, DjangoLDPAdmin)
djangoldp/check_integrity.py 0 → 100644
View file @ e97d33ad
'''
DjangoLDP Check Integrity
Usage `./manage.py check_integrity --help`
'''
from django.apps import apps
from django.conf import settings
from djangoldp.models import LDPSource
from urllib.parse import urlparse
import requests
# Helper command for argument type checking
def is_string(target):
if isinstance(target, str):
return target
return False
# Helper command to check status code of a target
def is_alive(target, status_code = 200):
return requests.get(target).status_code == status_code
# Add argument to the `check_integrity` command
def add_arguments(parser):
parser.add_argument(
"--ignore",
action="store",
default=False,
type=is_string,
help="Ignore any server, comma separated",
)
parser.add_argument(
"--ignore-faulted",
default=False,
nargs="?",
const=True,
help="Ignore eventual faulted",
)
parser.add_argument(
"--ignore-404",
default=False,
nargs="?",
const=True,
help="Ignore eventual 404",
)
parser.add_argument(
"--fix-faulted-resources",
default=False,
nargs="?",
const=True,
help="Fix faulted resources",
)
parser.add_argument(
"--fix-404-resources",
default=False,
nargs="?",
const=True,
help="Fix 404 resources",
)
parser.add_argument(
"--fix-offline-servers",
default=False,
nargs="?",
const=True,
help="Remove resources from offline servers",
)
# Define our own checks
def check_integrity(options):
models = apps.get_models()
ignored = set()
if(options["ignore"]):
for target in options["ignore"].split(","):
ignored.add(urlparse(target).netloc)
if(len(ignored) > 0):
print("Ignoring servers:")
for server in ignored:
print("- "+server)
resources = set()
resources_map = dict()
base_urls = set()
for model in models:
for obj in model.objects.all():
if hasattr(obj, "urlid"):
if(obj.urlid):
if(not obj.urlid.startswith(settings.BASE_URL)):
url = urlparse(obj.urlid).netloc
if(url not in ignored):
resources.add(obj.urlid)
resources_map[obj.urlid] = obj
base_urls.add(url)
if(len(base_urls) > 0):
print("Found "+str(len(resources_map))+" distant resources on "+str(len(models))+" models")
print("Servers that I have backlinks to:")
for server in base_urls:
print("- "+server)
else:
print("I don't have any backlink")
source_urls = set()
for source in LDPSource.objects.all():
source_urls.add(urlparse(source.urlid).netloc)
if(len(source_urls) > 0):
print("Servers that I'm allowed to get federated to:")
for server in source_urls:
print("- "+server)
else:
print("I'm not federated")
difference_servers = base_urls.difference(source_urls)
if(len(difference_servers) > 0):
print("Servers that I should not get aware of:")
for server in difference_servers:
print("- "+server)
# Handle faulted resources
if(not options["ignore_faulted"]):
faulted_resources = set()
for server in difference_servers:
for resource in resources:
if(urlparse(resource).netloc in server):
faulted_resources.add(resource)
if(len(faulted_resources) > 0):
print("Resources in fault:")
for resource in faulted_resources:
print("- "+resource)
else:
print("No resource are in fault")
if(options["fix_faulted_resources"]):
for resource in faulted_resources:
try:
resources_map[resource].delete()
except:
pass
print("Fixed faulted resources")
else:
print("Fix them with `./manage.py check_integrity --fix-faulted-resources`")
else:
print("I accept datas for every of those servers")
# Handle 404 resources
if(not options["ignore_404"]):
resources_404 = set()
resources_servers_offline = set()
for resource in resources:
try:
if(is_alive(resource, 404)):
resources_404.add(resource)
except:
resources_servers_offline.add(resource)
if(len(resources_404) > 0):
print("Faulted resources, 404:")
for resource in resources_404:
print("- "+resource)
if(options["fix_404_resources"]):
for resource in resources_404:
try:
resources_map[resource].delete()
except:
pass
print("Fixed 404 resources")
else:
print("Fix them with `./manage.py check_integrity --fix-404-resources`")
if(len(resources_servers_offline) > 0):
print("Faulted resources, servers offline:")
for resource in resources_servers_offline:
print("- "+resource)
if(options["fix_offline_servers"]):
for resource in resources_servers_offline:
try:
resources_map[resource].delete()
except:
pass
print("Fixed resources on offline servers")
else:
print("Fix them with `./manage.py check_integrity --fix-offline-servers`")
else:
print("No 404 in known resources")
\ No newline at end of file
djangoldp/cli.py
View file @ e97d33ad
......@@ -14,14 +14,14 @@ from . import __version__
@click.version_option(__version__)
def main():
"""DjangoLDP"""
"""DjangoLDP CLI"""
@main.command()
@click.argument('name', nargs=1, required=False)
def startproject(name):
def initserver(name):
"""Start a DjangoLDP project."""
"""Start a DjangoLDP server."""
try:
# use directly pwd
......@@ -37,15 +37,19 @@ def startproject(name):
# wrap the default django-admin startproject command
# this call import django settings and configure it
# see: https://docs.djangoproject.com/fr/1.11/topics/settings/#calling-django-setup-is-required-for-standalone-django-usage
management.call_command('startproject', name, directory, template=template, name=['config.yml'])
# see: https://docs.djangoproject.com/fr/2.2/topics/settings/#calling-django-setup-is-required-for-standalone-django-usage
# see: https://github.com/django/django/blob/stable/2.2.x/django/core/management/templates.py#L108
# fix: in 2.2 gabarit files options has been renamed: https://github.com/django/django/blob/stable/2.2.x/django/core/management/templates.py#L53
management.call_command('startproject', name, directory, template=template, files=['settings.yml'])
except FileExistsError:
click.echo(f'Error: the folder {directory} already exists')
sys.exit(1)
except CommandError as e:
click.echo(f'Error: {e}')
directory.rmdir()
sys.exit(1)
@main.command()
@click.argument('name', nargs=1)
......@@ -68,9 +72,11 @@ def startpackage(name):
except FileExistsError:
click.echo(f'Error: the folder {directory} already exists')
sys.exit(1)
except CommandError as e:
click.echo(f'Error: {e}')
sys.exit(1)
@main.command()
def install():
......@@ -79,12 +85,12 @@ def install():
try:
# load dependencies from config file
path = Path.cwd() / 'config.yml'
path = Path.cwd() / 'settings.yml'
with open(path, 'r') as f:
dependencies = yaml.safe_load(f).get('dependencies', [])
# install them by calling pip command
cmd = [sys.executable, "-m", "pip", "install"]
cmd = [sys.executable, "-m", "pip", "install", "--upgrade"]
try:
cmd.extend(dependencies)
subprocess.run(cmd).check_returncode()
......@@ -93,14 +99,19 @@ def install():
click.echo('No dependency to install')
except FileNotFoundError:
click.echo('Config error: no config.yml file in this directory')
click.echo('Config error: no settings.yml file in this directory')
sys.exit(1)
except subprocess.CalledProcessError as e:
click.echo(f'Installation error: {e}')
sys.exit(1)
@main.command()
def configure():
@click.option('--with-admin', 'admin', help='Create an administrator user with email.')
@click.option('--email', help='Provide an email for administrator.')
@click.option('--with-dummy-admin', 'dummy_admin', is_flag=True, help='Create a default "admin" user.')
def configure(admin, dummy_admin, email):
"""Configure the project."""
......@@ -108,12 +119,20 @@ def configure():
# shortcut to the djangoldp.management command
path = str(Path.cwd() / 'manage.py')
cmd = [sys.executable, path, 'configure']
if admin:
if not email:
click.echo('Error: missing email for admin user')
return
cmd.extend(['--with-admin', admin, '--email', email])
elif dummy_admin:
cmd.append('--with-dummy-admin')
subprocess.run(cmd).check_returncode()
click.echo('Confguration done!')
click.echo('Configuration done!')
except subprocess.CalledProcessError as e:
click.echo(f'Configuration error: {e}')
sys.exit(1)
@main.command()
......@@ -129,3 +148,4 @@ def runserver():
except subprocess.CalledProcessError as e:
click.echo(f'Execution error: {e}')
sys.exit(1)
djangoldp/conf/default_settings.py 0 → 100644
View file @ e97d33ad
"""This module override some of default django configuration from the global settings."""
from django.conf.global_settings import *
####################
# LDP #
####################
LDP_RDF_CONTEXT = 'https://cdn.startinblox.com/owl/context.jsonld'
MAX_ACTIVITY_RESCHEDULES = 3
DEFAULT_BACKOFF_FACTOR = 1
DEFAULT_ACTIVITY_DELAY = 0.2
DEFAULT_REQUEST_TIMEOUT = 10
LDP_INCLUDE_INNER_PERMS = False
####################
# CORE #
####################
# https://en.wikipedia.org/wiki/List_of_tz_zones_by_name (although not all
# systems may support all possibilities). When USE_TZ is True, this is
# interpreted as the default user time zone.
TIME_ZONE = 'UTC'
# If you set this to True, Django will use timezone-aware datetimes.
USE_TZ = True
# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'en-us'
# Database connection info. If left empty, will default to the dummy backend.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': 'db.sqlite3',
}
}
# List of strings representing installed apps.
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'djangoldp',
'guardian'
]
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
# Default server url
SITE_URL = 'http://localhost:8000'
BASE_URL = SITE_URL
# Default URL conf
ROOT_URLCONF = 'djangoldp.urls'
# Absolute filesystem path to the directory that will hold user-uploaded files.
# Example: "/var/www/example.com/media/"
MEDIA_ROOT = ''
# URL that handles the media served from MEDIA_ROOT.
# Examples: "http://example.com/media/", "http://media.example.com/"
MEDIA_URL = '/media/'
# Absolute path to the directory static files should be collected to.
# Example: "/var/www/example.com/static/"
STATIC_ROOT = None
# URL that handles the static files served from STATIC_ROOT.
# Example: "http://example.com/static/", "http://static.example.com/"
STATIC_URL = '/static/'
# ETAGS config
USE_ETAGS = True
# Default X-Frame-Options header value
X_FRAME_OPTIONS = 'DENY'
# The Python dotted path to the WSGI application that Django's internal server
# (runserver) will use. If `None`, the return value of
# 'django.core.wsgi.get_wsgi_application' is used, thus preserving the same
# behavior as previous versions of Django. Otherwise this should point to an
# actual WSGI application object.
WSGI_APPLICATION = 'server.wsgi.application'
##############
# MIDDLEWARE #
##############
# List of middleware to use. Order is important; in the request phase, these
# middleware will be applied in the order given, and in the response
# phase the middleware will be applied in reverse order.
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'djangoldp.middleware.AllowRequestedCORSMiddleware',
'django.middleware.gzip.GZipMiddleware',
'django_brotli.middleware.BrotliMiddleware'
]
##################
# REST FRAMEWORK #
##################
REST_FRAMEWORK = {
'DEFAULT_SCHEMA_CLASS': 'drf_spectacular.openapi.AutoSchema',
'DEFAULT_PAGINATION_CLASS': 'djangoldp.pagination.LDPPagination'
}
###################
# DRF SPECTACULAR #
###################
ENABLE_SWAGGER_DOCUMENTATION = False
SPECTACULAR_SETTINGS = {
'TITLE': 'DjangoLDP Based API Description',
'DESCRIPTION': 'Here you will find the list of all endpoints available on your Djangoldp-based server instance and\
the available methods, needed parameters and requests examples. The list of available endpoints depend on your instance configuration\
especially the list of bloxes and associated django models activated.',
'VERSION': '2.1.37',
'SERVE_INCLUDE_SCHEMA': False
}
############
# SESSIONS #
############
# Same site policy
DCS_SESSION_COOKIE_SAMESITE = 'none'
##################
# AUTHENTICATION #
##################
AUTHENTICATION_BACKENDS = ['django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend']
OIDC_ACCESS_CONTROL_ALLOW_HEADERS = 'Content-Type, if-match, accept, authorization, DPoP, cache-control, pragma, prefer'
# The minimum number of seconds a password reset link is valid for
PASSWORD_RESET_TIMEOUT = 60 * 60 * 24 * 3
DISABLE_LOCAL_OBJECT_FILTER = False
GUARDIAN_AUTO_PREFETCH = True
DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
djangoldp/conf/global_settings.py deleted 100644 → 0
View file @ 68676692
This diff is collapsed.
djangoldp/conf/ldpsettings.py 0 → 100644
View file @ e97d33ad
import os
import sys
import yaml
import logging
from django.core.exceptions import ImproperlyConfigured
from django.conf import settings as django_settings
from pathlib import Path
from collections import OrderedDict
from typing import Iterable
from importlib import import_module
from . import default_settings
logger = logging.getLogger(__name__)
def configure(filename='settings.yml'):
"""Helper function to configure django from LDPSettings."""
yaml_config = None
try:
with open(filename, 'r') as f:
yaml_config = yaml.safe_load(f)
except FileNotFoundError:
logger.info('Starting project without configuration file')
# ref: https://docs.djangoproject.com/fr/2.2/topics/settings/#custom-default-settings
ldpsettings = LDPSettings(yaml_config)
django_settings.configure(ldpsettings)
class LDPSettings(object):
"""Class managing the DjangoLDP configuration."""
def __init__(self, config):
"""Build a Django Setting object from a dict."""
if django_settings.configured:
raise ImproperlyConfigured('Settings have been configured already')
self._config = config
self._explicit_settings = set(default_settings.__dict__.keys()) #default settings are explicit
self._settings = self.build_settings()
def build_settings(self, extend=['INSTALLED_APPS', 'MIDDLEWARE']):
"""
Look for the parameters in multiple places.
Each step overrides the value of the previous key found. Except for "extend" list. Those value must be lists and all values found are added to these lists without managing duplications.
Resolution order of the configuration:
1. Core default settings
2. Packages settings
3. Code from a local settings.py file
4. YAML config file
"""
# helper loop
def update_with(config):
for setting, value in config.items():
self._explicit_settings.add(setting)
if setting in extend:
settings[setting].extend(value)
elif not setting.startswith('_'):
settings.update({setting: value})
# start from default core settings
settings = default_settings.__dict__.copy()
logger.debug(f'Building settings from core defaults')
# INSTALLED_APPS starts empty
settings['INSTALLED_APPS'] = []
# look settings from packages in the order they are given (local overrides installed)
for pkg in self.DJANGOLDP_PACKAGES:
# FIXME: There is something better to do here with the sys.modules path
try:
# override with values from installed package
mod = import_module(f'{pkg}.djangoldp_settings')
update_with(mod.__dict__)
logger.debug(f'Updating settings from installed package {pkg}')
except ModuleNotFoundError:
pass
try:
# override with values from local package
mod = import_module(f'{pkg}.{pkg}.djangoldp_settings')
update_with(mod.__dict__)
logger.debug(f'Updating settings from local package {pkg}')
except ModuleNotFoundError:
pass
# look in settings.py file in directory
try:
mod = import_module('settings')
update_with(mod.__dict__)
logger.debug(f'Updating settings from local settings.py file')
except ModuleNotFoundError:
pass
# look in YAML config file 'server' section
try:
conf = self._config.get('server', {})
update_with(conf)
logger.debug(f'Updating settings with project config')
except KeyError:
pass
# In the end adds the INSTALLED_APPS from the core
settings['INSTALLED_APPS'].extend(getattr(default_settings,'INSTALLED_APPS'))
return settings
@property
def DJANGOLDP_PACKAGES(self):
"""Returns the list of LDP packages configured."""
pkg = self._config.get('ldppackages', [])
return [] if pkg is None else pkg
@property
def INSTALLED_APPS(self):
"""Return the installed apps and the LDP packages."""
# get ldp packages (they are django apps)
apps = self.DJANGOLDP_PACKAGES.copy()
# add the default apps
apps.extend(self._settings['INSTALLED_APPS'])
# As settings come from different origins duplicuation is likeliy to happen
return list(OrderedDict.fromkeys(apps))
def __getattr__(self, param):
"""Return the requested parameter from cached settings."""
if param.startswith('_') or param.islower():
# raise the django exception for inexistent parameter
raise AttributeError(f'"{param}" is not compliant to django settings format')
try:
return self._settings[param]
except KeyError:
# raise the django exception for inexistent parameter
raise AttributeError(f'no "{param}" parameter found in settings')
def is_overridden(self, setting):
return setting in self._explicit_settings
\ No newline at end of file
djangoldp/conf/package_template/app_name/djangoldp_settings.py-tpl
View file @ e97d33ad
......@@ -6,3 +6,6 @@ MYPACKAGE_VAR = 'MY_DEFAULT_VAR'
# register an extra middleware
MIDDLEWARE = []
# register an extra installed app
INSTALLED_APPS = []
djangoldp/conf/package_template/app_name/djangoldp_urls.py-tpl
View file @ e97d33ad
from django.conf.urls import url
from django.urls import path
from .models import Dummy
from .views import DummyViewset
urlpatterns = [
url(r'^dummy/', DummyViewset.urls(model_prefix="dummy", model=Dummy))
path('dummy/', DummyViewset.urls(model_prefix="dummy", model=Dummy))
]
djangoldp/conf/package_template/app_name/models.py-tpl
View file @ e97d33ad
from django.db import models
from djangoldp.models import Model
class Dummy(Model):
......