BGE: generic python callback list + replace KX_PythonSeq.

I made this patch to declared a python list without converting all elements in python object (too slow) or use a CListValue which required CValue items (too expensive in memory).  In the case of a big list of points like a collision contacts points list, to use a CListValue we must implement a new class based on CValue for 3D vector to create a python proxy even if mathutils do it perfectly, we must also convert all points (frequently ~100 points) when fill the CListValue even if the list is not used (in the case of the collision callback). The easy way is to use callback (it doesn't worth to do an inheritance) which convert the item in PyObject only during an acces.
5 callbacks are used :
- Check if the list is valid = allow acces (like PyObjectPlus.invalid)
- Get the list size
- Get an item in the list by index.
- Get an item name in the list by index (used for operator `list["name"]`)
- Set an item in the list at the index position.
All of these callback take as first argument the client instance.
Why do we use a void * for the client instance ? : In KX_PythonInitTypes.cpp we have to initialize each python inherited class, if we use a template (the only other way) we must add this class each time we use a new type with in KX_PythonInitTypes.cpp

To check if the list can be accessed from python by the user, we check if the python proxy,  which is the `m_base` member, is still a valid proxy like in PyObjectPlus. But we can use a callback for more control of user access (e.g a list of collision point invalidate a frame later, in this case no real python owner).

This python list is easily defined with :
```
CPythonCallBackList(
void *client, // The client instance
PyObject *base, // The python instance which owned this list, used to know if the list is valid (like in KX_PythonSeq)
bool (*checkValid)(void *), // A callback to check if this list is till valid (optional)
int (*getSize)(void *), // A callback to get size
PyObject *(*getItem)(void *, int), // A callback to get an item
const char *(*getItemName)(void *, int), // A callback to get an item name (optional) use for acces by string key
bool (*setItem)(void *, int, PyObject *) // A callback to set an item (optional)
)
```
To show its usecase i replaced the odd KX_PythonSeq, it modify KX_Gameobject.sensors/controllers/actuators, SCA_IController.sensors/actuators and BL_ArmatureObject.constraints/channels.

Example : {F245193}, See message in console, press R to erase the object and see invalid proxy error message.

Reviewers: brita_, #game_python, youle, campbellbarton, moguri, agoose77, sergey

Reviewed By: campbellbarton, moguri, agoose77, sergey

Subscribers: sergey

Projects: #game_engine

Differential Revision: https://developer.blender.org/D1363

3 files changed, 536 insertions, 0 deletions

diff --git a/source/gameengine/Expressions/CMakeLists.txt b/source/gameengine/Expressions/CMakeLists.txt
index 6ab4d3fdacc..9c563a46ea2 100644
--- a/source/gameengine/Expressions/CMakeLists.txt
+++ b/source/gameengine/Expressions/CMakeLists.txt
@@ -54,6 +54,7 @@ set(SRC
 	intern/StringValue.cpp
 	intern/Value.cpp
 	intern/VectorValue.cpp
+	intern/ListWrapper.cpp
 
 	EXP_BoolValue.h
 	EXP_ConstExpr.h
@@ -75,6 +76,8 @@ set(SRC
 	EXP_Value.h
 	EXP_VectorValue.h
 	EXP_VoidValue.h
+	EXP_ListWrapper.h
+
 )
 
 if(WITH_PYTHON)
diff --git a/source/gameengine/Expressions/EXP_ListWrapper.h b/source/gameengine/Expressions/EXP_ListWrapper.h
new file mode 100644
index 00000000000..e4c97691acb
--- /dev/null
+++ b/source/gameengine/Expressions/EXP_ListWrapper.h
@@ -0,0 +1,109 @@
+/*
+ * ***** BEGIN GPL LICENSE BLOCK *****
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software Foundation,
+ * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * Contributor(s): Porteries Tristan.
+ *
+ * ***** END GPL LICENSE BLOCK *****
+ */
+
+/** \file EXP_ListWrapper.h
+ *  \ingroup expressions
+ */
+
+#ifdef WITH_PYTHON
+
+#ifndef __EXP_LISTWRAPPER_H__
+#define __EXP_LISTWRAPPER_H__
+
+#include "EXP_Value.h"
+
+class CListWrapper : public CValue  
+{
+	Py_Header
+private:
+	/** The client instance passed as first argument of each callback.
+	 * We use a void * instead of a template to avoid to declare this class
+	 * for each use in KX_PythonInitTypes.
+	 */
+	void *m_client;
+
+	// The python object which owned this list.
+	PyObject *m_base;
+
+	/// Returns true if the list is still valid, else each call will raise an error.
+	bool (*m_checkValid)(void *);
+
+	/// Returns the list size.
+	int (*m_getSize)(void *);
+
+	/// Returns the list item for the giving index.
+	PyObject *(*m_getItem)(void *, int);
+
+	/// Returns name item for the giving index, used for python operator list["name"].
+	const char *(*m_getItemName)(void *, int);
+
+	/// Sets the nex item to the index place, return false when failed item conversion.
+	bool (*m_setItem)(void *, int, PyObject *);
+
+public:
+	CListWrapper(void *client,
+						PyObject *base,
+						bool (*checkValid)(void *),
+						int (*getSize)(void *),
+						PyObject *(*getItem)(void *, int),
+						const char *(*getItemName)(void *, int),
+						bool (*setItem)(void *, int, PyObject *));
+	~CListWrapper();
+
+	/// \section Python Interface
+	bool CheckValid();
+	int GetSize();
+	PyObject *GetItem(int index);
+	const char *GetItemName(int index);
+	bool SetItem(int index, PyObject *item);
+	bool AllowSetItem();
+	bool AllowGetItemByName();
+
+	/// \section CValue Inherited Functions.
+	virtual const STR_String &GetText();
+	virtual void SetName(const char *name);
+	virtual STR_String &GetName();
+	virtual CValue *GetReplica();
+	virtual CValue *Calc(VALUE_OPERATOR op, CValue *val);
+	virtual CValue *CalcFinal(VALUE_DATA_TYPE dtype, VALUE_OPERATOR op, CValue *val);
+	virtual double GetNumber();
+	virtual int GetValueType();
+	virtual PyObject *py_repr();
+
+	// Python list operators.
+	static PySequenceMethods py_as_sequence;
+	// Python dictionnary operators.
+	static PyMappingMethods py_as_mapping;
+
+	static Py_ssize_t py_len(PyObject *self);
+	static PyObject *py_get_item(PyObject *self, Py_ssize_t index);
+	static int py_set_item(PyObject *self, Py_ssize_t index, PyObject *value);
+	static PyObject *py_mapping_subscript(PyObject *self, PyObject *key);
+	static int py_mapping_ass_subscript(PyObject *self, PyObject *key, PyObject *value);
+	static int py_contains(PyObject *self, PyObject *key);
+
+	KX_PYMETHOD_VARARGS(CListWrapper, Get);
+};
+
+#endif // __EXP_LISTWRAPPER_H__
+
+#endif // WITH_PYTHON
diff --git a/source/gameengine/Expressions/intern/ListWrapper.cpp b/source/gameengine/Expressions/intern/ListWrapper.cpp
new file mode 100644
index 00000000000..db1518a4388
--- /dev/null
+++ b/source/gameengine/Expressions/intern/ListWrapper.cpp
@@ -0,0 +1,424 @@
+/*
+ * ***** BEGIN GPL LICENSE BLOCK *****
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software Foundation,
+ * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * Contributor(s): Porteries Tristan.
+ *
+ * ***** END GPL LICENSE BLOCK *****
+ */
+
+/** \file ListWrapper.cpp
+ *  \ingroup expressions
+ */
+
+#ifdef WITH_PYTHON
+
+#include "EXP_ListWrapper.h"
+
+static STR_String pythonGeneratorList = "ListWrapper";
+
+CListWrapper::CListWrapper(void *client,
+						   PyObject *base,
+						   bool (*checkValid)(void *),
+						   int (*getSize)(void *),
+						   PyObject *(*getItem)(void *, int),
+						   const char *(*getItemName)(void *, int),
+						   bool (*setItem)(void *, int, PyObject *))
+:m_client(client),
+m_base(base),
+m_checkValid(checkValid),
+m_getSize(getSize),
+m_getItem(getItem),
+m_getItemName(getItemName),
+m_setItem(setItem)
+{
+	// Incref to always have a existing pointer.
+	Py_INCREF(m_base);
+}
+
+CListWrapper::~CListWrapper()
+{
+	Py_DECREF(m_base);
+}
+
+bool CListWrapper::CheckValid()
+{
+	if (m_base && !BGE_PROXY_REF(m_base)) {
+		return false;
+	}
+	return m_checkValid ? (*m_checkValid)(m_client) : true;
+}
+
+int CListWrapper::GetSize()
+{
+	return (*m_getSize)(m_client);
+}
+
+PyObject *CListWrapper::GetItem(int index)
+{
+	return (*m_getItem)(m_client, index);
+}
+
+const char *CListWrapper::GetItemName(int index)
+{
+	return (*m_getItemName)(m_client, index);
+}
+
+bool CListWrapper::SetItem(int index, PyObject *item)
+{
+	return (*m_setItem)(m_client, index, item);
+}
+
+bool CListWrapper::AllowSetItem()
+{
+	return m_setItem != NULL;
+}
+
+bool CListWrapper::AllowGetItemByName()
+{
+	return m_getItemName != NULL;
+}
+
+// ================================================================
+
+const STR_String &CListWrapper::GetText()
+{
+	return pythonGeneratorList;
+}
+
+void CListWrapper::SetName(const char *name)
+{
+}
+
+STR_String &CListWrapper::GetName()
+{
+	return pythonGeneratorList;
+}
+
+CValue *CListWrapper::GetReplica()
+{
+	return NULL;
+}
+
+CValue *CListWrapper::Calc(VALUE_OPERATOR op, CValue *val)
+{
+	return NULL;
+}
+
+CValue *CListWrapper::CalcFinal(VALUE_DATA_TYPE dtype, VALUE_OPERATOR op, CValue *val)
+{
+	return NULL;
+}
+
+double CListWrapper::GetNumber()
+{
+	return -1;
+}
+
+int CListWrapper::GetValueType()
+{
+	return -1;
+}
+
+// We convert all elements to python objects to make a proper repr string.
+PyObject *CListWrapper::py_repr()
+{
+	if (!CheckValid()) {
+		PyErr_SetString(PyExc_SystemError, "CListWrapper : repr, " BGE_PROXY_ERROR_MSG);
+		return NULL;
+	}
+
+	PyObject *py_proxy = GetProxy();
+	PyObject *py_list = PySequence_List(py_proxy);
+	PyObject *py_string = PyObject_Repr(py_list);
+	Py_DECREF(py_list);
+	Py_DECREF(py_proxy);
+	return py_string;
+}
+
+
+Py_ssize_t CListWrapper::py_len(PyObject *self)
+{
+	CListWrapper *list = (CListWrapper *)BGE_PROXY_REF(self);
+	// Invalid list.
+	if (!list->CheckValid()) {
+		PyErr_SetString(PyExc_SystemError, "len(CListWrapper), " BGE_PROXY_ERROR_MSG);
+		return 0;
+	}
+
+	return (Py_ssize_t)list->GetSize();
+}
+
+PyObject *CListWrapper::py_get_item(PyObject *self, Py_ssize_t index)
+{
+	CListWrapper *list = (CListWrapper *)BGE_PROXY_REF(self);
+	// Invalid list.
+	if (!list->CheckValid()) {
+		PyErr_SetString(PyExc_SystemError, "val = CListWrapper[i], " BGE_PROXY_ERROR_MSG);
+		return NULL;
+	}
+
+	int size = list->GetSize();
+
+	if (index < 0) {
+		index = size + index;
+	}
+	if (index < 0 || index >= size) {
+		PyErr_SetString(PyExc_IndexError, "CListWrapper[i]: List index out of range in CListWrapper");
+		return NULL;
+	}
+
+	PyObject *pyobj = list->GetItem(index);
+
+	return pyobj;
+}
+
+int CListWrapper::py_set_item(PyObject *self, Py_ssize_t index, PyObject *value)
+{
+	CListWrapper *list = (CListWrapper *)BGE_PROXY_REF(self);
+	// Invalid list.
+	if (!list->CheckValid()) {
+		PyErr_SetString(PyExc_SystemError, "CListWrapper[i] = val, " BGE_PROXY_ERROR_MSG);
+		return -1;
+	}
+
+	if (!list->AllowSetItem()) {
+		PyErr_SetString(PyExc_TypeError, "CListWrapper's item type doesn't support assignment");
+		return -1;
+	}
+
+	if (!value) {
+		PyErr_SetString(PyExc_TypeError, "CListWrapper doesn't support item deletion");
+		return -1;
+	}
+
+	int size = list->GetSize();
+
+	if (index < 0) {
+		index = size + index;
+	}
+	if (index < 0 || index >= size) {
+		PyErr_SetString(PyExc_IndexError, "CListWrapper[i]: List index out of range in CListWrapper");
+		return -1;
+	}
+
+	if (!list->SetItem(index, value)) {
+		return -1;
+	}
+	return 0;
+}
+
+PyObject *CListWrapper::py_mapping_subscript(PyObject *self, PyObject *key)
+{
+	CListWrapper *list = (CListWrapper *)BGE_PROXY_REF(self);
+	// Invalid list.
+	if (!list->CheckValid()) {
+		PyErr_SetString(PyExc_SystemError, "val = CListWrapper[key], " BGE_PROXY_ERROR_MSG);
+		return NULL;
+	}
+
+	if (PyIndex_Check(key)) {
+		Py_ssize_t index = PyLong_AsSsize_t(key);
+		return py_get_item(self, index);
+	}
+	else if (PyUnicode_Check(key)) {
+		if (!list->AllowGetItemByName()) {
+			PyErr_SetString(PyExc_SystemError, "CListWrapper's item type doesn't support access by key");
+			return NULL;
+		}
+
+		const char *name = _PyUnicode_AsString(key);
+		int size = list->GetSize();
+
+		for (unsigned int i = 0; i < size; ++i) {
+			if (strcmp(list->GetItemName(i), name) == 0) {
+				return list->GetItem(i);
+			}
+		}
+
+		PyErr_Format(PyExc_KeyError, "requested item \"%s\" does not exist", name);
+		return NULL;
+	}
+
+	PyErr_Format(PyExc_KeyError, "CListWrapper[key]: '%R' key not in list", key);
+	return NULL;
+}
+
+int CListWrapper::py_mapping_ass_subscript(PyObject *self, PyObject *key, PyObject *value)
+{
+	CListWrapper *list = (CListWrapper *)BGE_PROXY_REF(self);
+	// Invalid list.
+	if (!list->CheckValid()) {
+		PyErr_SetString(PyExc_SystemError, "val = CListWrapper[key], " BGE_PROXY_ERROR_MSG);
+		return -1;
+	}
+
+	if (!list->AllowSetItem()) {
+		PyErr_SetString(PyExc_TypeError, "CListWrapper's item type doesn't support assignment");
+		return -1;
+	}
+
+	if (PyIndex_Check(key)) {
+		Py_ssize_t index = PyLong_AsSsize_t(key);
+		return py_set_item(self, index, value);
+	}
+	else if (PyUnicode_Check(key)) {
+		if (!list->AllowGetItemByName()) {
+			PyErr_SetString(PyExc_SystemError, "CListWrapper's item type doesn't support access by key");
+			return -1;
+		}
+
+		const char *name = _PyUnicode_AsString(key);
+		int size = list->GetSize();
+
+		for (unsigned int i = 0; i < size; ++i) {
+			if (strcmp(list->GetItemName(i), name) == 0) {
+				if (!list->SetItem(i, value)) {
+					return -1;
+				}
+				return 0;
+			}
+		}
+
+		PyErr_Format(PyExc_KeyError, "requested item \"%s\" does not exist", name);
+		return -1;
+	}
+
+	PyErr_Format(PyExc_KeyError, "CListWrapper[key]: '%R' key not in list", key);
+	return -1;
+}
+
+int CListWrapper::py_contains(PyObject *self, PyObject *key)
+{
+	CListWrapper *list = (CListWrapper *)BGE_PROXY_REF(self);
+	// Invalid list.
+	if (!list->CheckValid()) {
+		PyErr_SetString(PyExc_SystemError, "val = CListWrapper[i], " BGE_PROXY_ERROR_MSG);
+		return -1;
+	}
+
+	if (!list->AllowGetItemByName()) {
+		PyErr_SetString(PyExc_SystemError, "CListWrapper's item type doesn't support access by key");
+		return -1;
+	}
+
+	if (!PyUnicode_Check(key)) {
+		PyErr_SetString(PyExc_SystemError, "key in list, CListWrapper: key must be a string");
+		return -1;
+	}
+
+	const char *name = _PyUnicode_AsString(key);
+	int size = list->GetSize();
+
+	for (unsigned int i = 0; i < size; ++i) {
+		if (strcmp(list->GetItemName(i), name) == 0) {
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
+PySequenceMethods CListWrapper::py_as_sequence = {
+	py_len, // sq_length
+	NULL, // sq_concat
+	NULL, // sq_repeat
+	py_get_item, // sq_item
+	NULL, // sq_slice
+	py_set_item, // sq_ass_item
+	NULL, // sq_ass_slice
+	(objobjproc)py_contains, // sq_contains
+	(binaryfunc) NULL, // sq_inplace_concat
+	(ssizeargfunc) NULL, // sq_inplace_repeat
+};
+
+PyMappingMethods CListWrapper::py_as_mapping = {
+	py_len, // mp_length
+	py_mapping_subscript, // mp_subscript
+	py_mapping_ass_subscript // mp_ass_subscript
+};
+
+PyTypeObject CListWrapper::Type = {
+	PyVarObject_HEAD_INIT(NULL, 0)
+	"CListWrapper", // tp_name
+	sizeof(PyObjectPlus_Proxy), // tp_basicsize
+	0, // tp_itemsize
+	py_base_dealloc, // tp_dealloc
+	0, // tp_print
+	0, // tp_getattr
+	0, // tp_setattr
+	0, // tp_compare
+	py_base_repr, // tp_repr
+	0, // tp_as_number
+	&py_as_sequence, // tp_as_sequence
+	&py_as_mapping, // tp_as_mapping
+	0, // tp_hash
+	0, // tp_call
+	0,
+	NULL,
+	NULL,
+	0,
+	Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
+	0,0,0,0,0,0,0,
+	Methods,
+	0,
+	0,
+	&CValue::Type,
+	0,0,0,0,0,0,
+	py_base_new
+};
+
+PyMethodDef CListWrapper::Methods[] = {
+	{"get", (PyCFunction)CListWrapper::sPyGet, METH_VARARGS},
+	{NULL, NULL} //Sentinel
+};
+
+PyAttributeDef CListWrapper::Attributes[] = {
+	{NULL} //Sentinel
+};
+
+/* Matches python dict.get(key, [default]) */
+PyObject *CListWrapper::PyGet(PyObject *args)
+{
+	char *name;
+	PyObject *def = Py_None;
+
+	// Invalid list.
+	if (!CheckValid()) {
+		PyErr_SetString(PyExc_SystemError, "val = CListWrapper[i], " BGE_PROXY_ERROR_MSG);
+		return NULL;
+	}
+
+	if (!AllowGetItemByName()) {
+		PyErr_SetString(PyExc_SystemError, "CListWrapper's item type doesn't support access by key");
+		return NULL;
+	}
+
+	if (!PyArg_ParseTuple(args, "s|O:get", &name, &def)) {
+		return NULL;
+	}
+
+	for (unsigned int i = 0; i < GetSize(); ++i) {
+		if (strcmp(GetItemName(i), name) == 0) {
+			return GetItem(i);
+		}
+	}
+
+	Py_INCREF(def);
+	return def;
+}
+
+#endif // WITH_PYTHON